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The Virginia Department of Education’s Virtual Virginia (VVA) program delivers more than 90 supplemental 
online courses to more than 10,000 middle and high school students each year. The department partnered 
with Regional Educational Laboratory (REL) Appalachia to examine the relationship between students’ 
participation in a subset of 31 VVA online courses and their performance on end-of-course Virginia Standards 
of Learning (SOL) and Advanced Placement (AP) exams during the 2015/16 and 2016/17 school years. Relying 
on administrative records available through the Virginia Longitudinal Data System, REL Appalachia researchers 
conducted descriptive analyses and regression analyses with statistical controls. After controlling for 
demographic and prior achievement differences, students in face-to-face courses were more likely to score 
higher and demonstrate proficiency on SOL and AP exams than students taking the same courses through VVA 
across most subject areas. Because the study design was correlational, additional research is needed to 
determine whether the difference in performance is due to the characteristics of the students in online 
courses, aspects of the VVA program itself, or some other factor such as students’ reasons for enrolling in the 
course or the supports available in their local schools. 


Why this study? 

Online learning has been a growing trend in K-12 education since the 1990s. During the 2014/15 school year, 
approximately 2.7 million students were enrolled in K-12 online learning programs in the United States as a 
supplement to their regular instruction (Gemin, Pape, Vashaw, & Watson, 2015). Five states—including Virginia— 
now require students to complete at least one online course before high school graduation (Herold, 2017). This 
growth is driven by diverse interests, including expanding the range of courses available to students, offering 
increased flexibility for students with scheduling conflicts or who are otherwise unable to attend regular school, 
providing credit recovery options for students who have failed courses, and reducing per-pupil costs (U.S. 
Department of Education, Office of Educational Technology, 2012; Wicks, 2010). 


State-operated programs are a key supplier of online courses to schools and districts (Gemin & Pape, 2016), 
Operating in 23 states and serving about 420,000 students with almost a million course enrollments in the 2017/18 
fiscal year (Digital Learning Collaborative, 2019). Virtual Virginia (VVA), a program of the Virginia Department of 
Education (VDOE), serves more than 10,000 middle and high school students per year. VVA offers access to more 
than 90 supplemental online courses for Virginia students who otherwise may not be able to take Advanced 
Placement (AP), languages, core academic, and elective courses because of a lack of highly qualified instructors, 
insufficient student enrollment for a local course, or scheduling conflicts in a school. Eligible students enroll in 
VVA courses through their local schools (box 1). Starting in the 2014/15 school year, VVA expanded to support a 


full-time enrollment model that offers all the core academic courses and electives required to earn a high school 
diploma.' The full-time program serves approximately 200 students. 


Even as more students take online courses, evidence on the effectiveness of online learning programs is still 
sparse, with few rigorous published studies of the effectiveness of online learning for K-12 students (U.S. 
Department of Education, Office of Planning, Evaluation, and Policy Development, 2010; Brodersen & Melluzzo, 
2017). Findings from two of the most methodologically rigorous experimental studies are mixed, suggesting that 
at least some students may be able to benefit from online courses under some circumstances. For example, one 
study looking at the effectiveness of online courses for credit recovery found that students in online credit 
recovery courses were less likely to recover credit and scored lower on an algebra posttest than students in face- 
to-face credit recovery courses (Heppen et al., 2016a). Another study focused on the effectiveness of online 
courses to expand course access found that offering online algebra | in grade 8 to algebra-ready students in schools 
that do not typically offer the course positively impacted students’ algebra achievement and subsequent course 
taking (Heppen et al., 2011). This indicates a potential benefit to offering virtual learning opportunities to increase 
course availability when the course is not available face-to-face, one of the stated goals of VDOE’s VVA program. 


With limited and inconclusive evidence on online learning in Virginia and nationwide, Virginia’s 2014 
Appropriation Act (Virginia General Assembly, 2014) directed VDOE to analyze and report student effectiveness 
data for online programs. In response, the Virginia Joint Legislative Audit and Review Commission (2015) 
recommended that VDOE conduct ongoing analyses comparing student outcomes in VVA and face-to-face courses 
to explore the quality of online programs and support fiscally responsible use of state resources. 


As virtual learning in Virginia expands, the Virginia Board of Education, state legislators, and members of VDOE’s 
Virtual Learning Advisory Committee seek evidence to inform policy, program, and funding decisions. VDOE 
partnered with Regional Educational Laboratory (REL) Appalachia to begin examining how student achievement 
in VVA courses compared to student achievement in face-to-face courses.” 


This study provides descriptive information regarding participation in VVA courses and examines associations 
between participation in VVA courses and student academic performance. The study design is correlational, which 
does not allow for definitive conclusions about the impact of VVA on student achievement. Additionally, the 
researchers had access to virtual and face-to-face course enrollments, but not course offerings and so were unable 
to examine the benefit of increasing course availability through virtual means. However, the results can be used 
in discussions of virtual learning at the state and local levels, and they lay the groundwork for additional study. 


Box 1. The Virtual Virginia program 
The following description of the Virtual Virginia model was adapted by the authors from the program website.? 


Curriculum. Teams of content area experts, online learning experts, and instructional designers build the web-based courses 
for Virtual Virginia (VVA). Course materials include readings, simulations, interactive practice, video files, and audio files. 
VVA’s courses are aligned with the Virginia Standards of Learning (SOLs), and VVA’s Advanced Placement (AP) courses have 
been approved by the College Board. 


1 Full-time enrollment is defined as simultaneous enrollment in five or more VVA courses. Students move in and out of the full-time 
enrollment program based on the number of courses they take at one time. Supplemental enrollment is defined as simultaneous 
enrollment in four or fewer VVA courses. 

? Before partnering with REL Appalachia, VDOE researchers examined the pass rates and end-of-course exam scores for students in VVA 
courses. Although not the focus of this study, students’ unadjusted performance in VVA courses is presented in the technical appendix 
(exhibit A19). 

3 Source: https://www.virtualvirginia.org/ 


Instructors. VVA has both full-time and adjunct instructors who are Virginia-licensed, highly qualified, have experience in 
distance learning, and reside in Virginia. VVA provides instructors with professional development training in specific content 
areas and best practices of online instruction. VVA instructors manage the online course (set the pace, score student work) 
and work with students individually and in small groups online and by telephone to facilitate discussions, answer questions, 
and provide customized instruction. 


Enrollment. Students in Virginia middle and high schools who meet the course prerequisites and have approval from the local 
school counselor may enroll in VVA courses through their schools. Recommendations concerning instructional placement of 
the student are the responsibility of the local school counselor. Across schools, counselors have varied discretion in 
determining which students are eligible for and could succeed in online courses. 


Access and pacing. Students move through VVA course material at the pace of their cohort/start date. Instructors establish 
benchmark due dates to ensure that students stay on pace to complete the course by the end of the term/school year, but 
students have flexibility between those due dates. Students do not have to take VVA courses in ‘real’ time and can access 
courses 24/7 from any Internet-accessible location. 


Local implementation conditions and supports. VDOE and VVA do not prescribe or require a specific program model. There 
is a great deal of variation in the reasons for, and ways in which, students, schools, and school divisions use VVA. For example, 
some students take VVA courses during the school day in a computer lab monitored by a facilitator, while others take VVA 
courses at home using a personal device. 


Exams and grading. VVA awards students a numerical average for each course they complete. The local school then assigns 
credit and the appropriate letter grade for this average based on the school’s grading scale. Students enrolled in VVA courses 
that include an SOL exam must take the test through their local school. VVA encourages but does not require students in VVA 
AP courses to take the corresponding AP exam. 


Costs. During the academic year, each Virginia public school may enroll up to 15 students per course with no enrollment fees. 
Local schools are responsible for ensuring that students in the program have access to student services, technology, 
textbooks, and other necessary materials at no cost. VVA summer session courses cost $375 per course. 


Research questions 


This study addressed two primary research questions to explore the relationship between students’ participation 
in VVA courses and their short-term academic success as measured by standardized test score outcomes: 


1. How do the demographic characteristics compare between students who enrolled in and 
completed Virtual Virginia courses and the standardized end-of-course exams and students 
who took the same courses and exams face to face? 


2. How do students who enrolled in and completed Virtual Virginia courses and the standardized 
end-of-course exam perform, compared to similar students who took the same courses and 
exams face to face? 


Answering these questions provides information on the extent to which student achievement is comparable for 
the two available instructional modes. Controlling for differences in demographics and prior achievement enables 
a comparison between similar students in each group. Key terms used in this report are defined below (box 2). 


Box 2. Key Terms 


Advanced Placement (AP) course. College-level courses developed by the College Board’s AP Program that high schools can 
choose to offer along with corresponding exams that are administered once a year. AP exams are scored on a scale of 1 to 5. 


Many U.S. colleges and universities grant students course credit, advanced course placement, or both for AP exam scores of 
3 or higher.* 


Core academic course. A course required to earn a high school diploma. 
Division. Local education agencies in Virginia, which are similar to school districts in some other states. 


Economically disadvantaged. A VDOE designation that identifies a student who meets any one or more of the following: is 
eligible for free or reduced-price meals, receives assistance from the Temporary Assistance for Needy Families program, is 
eligible for Medicaid, or is identified as either a migrant or experiencing homelessness. 


End-of-course exam. Term used throughout this report to refer to Advanced Placement (AP) exams and Virginia Standards 
of Learning (SOL) exams. 


Face-to-face course. Course taken in person in a traditional classroom setting. 
Full-time course enrollment model. Defined by VDOE as concurrent enrollment in five or more VVA courses. 


Gifted. A VDOE designation identifying a student who demonstrates high levels of accomplishment or who shows the 
potential for higher levels of accomplishment when compared with others of the same age, experience, or environment. The 
aptitudes and potential of gifted students for accomplishment are so outstanding that they require special programs to meet 
their education need. Each school division in Virginia establishes procedures for the identification of gifted students and for 
the delivery of services to those students, consistent with the Regulations Governing Educational Services for Gifted Students. 
Eligibility decisions must be based on multiple criteria which may include test scores. 


Locale classification. A general geographic indicator developed by the National Center for Education Statistics Education 
Demographic and Geographic Estimates program that categorizes U.S. territory into four types of areas: city, suburban, town, 
and rural. 


Online course. Any course offered in which instruction and content are delivered primarily over the Internet; synonymous 
with e-course, virtual course, and cyber course. 


Standards of Learning (SOLs). The minimum expectations for what Virginia public school students should know and be able 
to do at the end of each grade or course in English, mathematics, science, history/social science, and other subjects. SOL 
exams measure students’ success in meeting the Board of Education’s expectations for learning and achievement. Student 
performance is graded on a scale of 0-600, with 400 representing the minimum level of acceptable proficiency. 


Subject area. Synonymous with content area or domain. Examples include English language arts, mathematics, and science. 


Substantive importance. An effect size—a standardized measure of the magnitude of the difference in outcomes or 
characteristics between groups—equal to or greater than 0.25. 


Supplemental course enrollment model. Defined by the Virginia Department of Education as concurrent enrollment in four 
or fewer Virtual Virginia (VVA) courses. 


Study population 

To compare course-related test scores of students taking the same course through face-to-face instruction and 
VVA, researchers constrained the study population to those courses that had an end-of-course standardized test 
and to those students who enrolled in and completed the course and who also took the end-of-course exam. In 
2015/16 and 2016/17, about 8,000 students enrolled in a VVA course that had an associated end-of-course exam 
(exhibit 1). About 3,800 (48 percent) of those students completed the course and took the end-of-course test. In 
contrast, about 1.41 million students enrolled in the same courses face to face, and about 1.14 million (81 percent) 


4 Source: https://apstudents.collegeboard.org/about-ap-scores 


of those students competed the course and took the end-of-course exam. Although students’ reasons for not 
completing the course or the end-of-course exam in either instructional mode are unknown, these patterns are 
consistent with prior research findings that online courses often have a higher noncompletion rate than face-to- 
face courses (Bawa, 2016; Freidhoff, 2016).° 


Exhibit 1. The population for the study comprised Virginia students who enrolled in and completed the 
course, and took the associated end-of-course exam during the 2015/16 and 2016/17 school years 


Virtual Virginia students Face to face 


Took the end-of- 
The study course exam 
population N= 1,142,100 


* Enrollment in a Virtual Virginia course reflects a record of a course account created for the student. In some cases, schools upload entire rosters of students, 
often not knowing which students plan to enroll and which students do not. It is not possible to determine from the available administrative data whether 
individual students who have a course account were enrolled or part of a larger roster of students who may or may not enroll in the course. As such, these 
counts may over-estimate the number of students enrolled in VVA courses. 

» The Virginia Longitudinal Data System does not include an indicator for course completion. The visualization depicts a decrease in the number of students 
(N) between course completion and exam completion, although the exact change in numbers, if any, cannot be determined from the available data. It is 
possible that the number of students who completed the course is equal to the number who took the end-of-course exam. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Enrolled? in 
a course with an end-of-course 
exam 


N= 1,407,755 


Completed the course® 


Study design 

REL Appalachia researchers reviewed available administrative data sources (box 3) to determine the most rigorous 
study design possible using existing data. In consultation with VDOE and the Institute for Education Sciences, 
researchers selected a correlational design using regression analyses. Those analyses sought to take into account 
factors related to students and schools that might influence differences in exam scores between VVA and face-to- 
face courses. However, regression could only account for factors for which data were available in the Virginia 
Longitudinal Data System (VLDS), which included prior test scores, student demographics, and school 
characteristics. Even after accounting for these factors, there are likely other important differences between 
students in the VVA and face-to-face conditions that this study did not address. For example, the current study 
did not include data on the following: 


5 The reasons for course “withdrawals” prior to test completion are very limited in the data that VDOE collects, and the information that 
is recorded is too unreliable to report. For this reason, we do not provide any additional information on the group of students who 
enrolled in a VVA course but did not complete the associated end-of-course SOL or AP exam. 


e Students’ reasons for enrolling in a VVA course, such as to take a course not offered at their local school, to recover credit 
for a course they previously failed in a face-to-face setting,° or because they were homebound.’ 


e Student characteristics such as self-motivation, self-discipline, and time management skills. 


e Differences in school-level implementations of VVA, including the kinds of students whom school counselors identify to 
take online courses and the supports available to students. 


e Students’ reasons for taking or not taking end-of-course exams, nor the reason fewer of the VVA students completed end- 
of-course exams than their face-to-face course peers. 


Research has shown these factors may also be related to student persistence and performance in online courses 
(Bawa, 2016; Hung, Hsu, & Rice, 2012; Kim, Park, & Cozart, 2014; Liu & Cavanaugh, 2011; Rice, 2006; Roblyer, 
Davis, Mills, Marshall, & Pape, 2008). Therefore, the correlations reported in this study may be attributable, in 
part, to these unmeasured factors. 


Box 3. Data sources, population, and methods 


For details on the data sources, population, methodology, and results, see Comparing the Achievement of Students in Virtual 
Virginia and Face-to-Face Courses: Report Technical Appendices (Mislevy, Schmidt, Puma, Ezekoye, & Saucedo 2020). 


Data sources. The Virginia Department of Education (VDOE) provided the data for these analyses through the Virginia 
Longitudinal Data System. The de-identified administrative records were at the student level and included student 
demographic variables, online and face-to-face course records, and achievement test scores. The dataset also included 
information on the student’s school of record, which researchers used to link in the locale classification from the National 
Center for Education Statistics’ Common Core of Data and the percentage of the region in which the school was located that 
was rural from the United States Census. 


Population. The study population consisted of all Virginia public school students in grades 8-12 who enrolled in courses with 
either SOL or AP exams during the 2015/16 and 2016/17 school years.® This comprised a total of 24 AP courses and seven 
core academic courses with SOL exams.? To be included in the study population, students needed to complete the end-of- 
course exam. 


Because no significant changes were made to the SOL or AP exams or to the online courses between 2015/16 and 2016/17, 
researchers aggregated data across the two school years to increase the available number of online students. Researchers 
also combined exam scores across courses in the same subject area. For example, algebra |, geometry, and algebra II SOL 
exam scores contributed to the “mathematics SOL” outcome. 


© The VLDS does not include an indicator for credit recovery courses, and observance of a student taking a course more than once does not 
systematically and reliably identify prior course failure. It is possible for a student to pass an end-of-course assessment and fail the course, 
and it is also possible for a student to fail an end-of-course assessment and pass the course (VDOE, 2009). 

7The VLDS does not include an indicator for homebound students. 

8 2015/16 was the first year VVA began tracking students’ unique state testing identifiers in its student information system, the linking 
variable needed to link VVA records to VDOE administrative records. 

° Although students in Virginia are also required to pass an SOL exam in history or social science, this study focuses on math, science, and 
English language arts core academic courses because they were the focus of VDOE’s prior analysis of unadjusted outcomes (analyses that 
did not include statistical controls for student demographic characteristics, prior achievement, or other factors). VDOE researchers 
excluded history/social science courses for several reasons. History is no longer a consideration for state accreditation and has a 
diminished presence in federal accountability. Additionally, VDOE recently reduced the number of verified credits required in history to 1, 
which has in turn reduced testing in history. Instead, history has been an important piece of VDOE's focus on developing and 
implementing performance-based assessments in lieu of standardized testing. Lastly, unlike the other subject areas, testing in history is 
not always sequential and ordered. 


Methodology. Two methodological approaches were used to address the research question. First, descriptive statistics were 
used to examine the characteristics of students who enrolled in and completed the end-of-course exam through VVA or in 
comparable face-to-face courses. 


Second, regression analysis was used to examine the relationship between VVA and SOL and AP exam scores while adjusting 
for the influence of other available student characteristics. Researchers included statistical controls for students’ grade level, 
race, gender, rurality, and economic status, as well as their prior test scores. 


The regressions pooled exam scores across courses within each subject area. Because these exams were not scored on the 
same scale,’ researchers used an accepted transformation to create a common outcome metric known as z-scores.*? 


Researchers also used regression with statistical controls to examine the relationship between taking the course through VVA 
and demonstrating proficiency on SOL and AP exams. For these analyses, proficiency was defined as achieving a score of 400 
or above on an SOL exam or 3 or above on an AP exam. 


REL Appalachia researchers used the criterion set by the What Works Clearinghouse (WWC, 2017) to determine the 
substantive importance of the differences in outcomes or characteristics between the students in VVA and face-to-face 
courses: an effect size—a standardized measure of the magnitude of the difference—of 0.25 or greater.?? 


Findings 


Students who enrolled in Virtual Virginia online courses represented a small fraction of all students 
who completed the associated end-of-course exams virtually or face-to-face. 


Face-to-face instruction was by far the more common mode of instruction for the courses and students in the 
study. In the population, 426 students were enrolled in VVA core academic courses and completed SOL exams, 
and 3,377 students were enrolled in VVA AP courses and completed AP exams during the 2015/16 and 2016/17 
school years (exhibit 2). VVA comprised just 0.05 percent of the total population in core academic courses, and 2 
percent of the total population in AP courses. 


10 Unlike the grade 3-8 SOL exams, which are vertically equated—that is, scored on the same scale across grades to allow for the direct 
comparison of performance across grade levels—the end-of-course SOLs that are the focus of this study are not equated to provide 
comparable scores across courses (VDOE, 2019). For all SOL exams, a score of 400 represents the minimum level of acceptable 
proficiency. However, outside this cut point, a score of 430 on the end-of-course exam for one course (for example, algebra |) may not be 
comparable to a score of 430 on the end-of-course exam for another course (for example, geometry). 

4 For each end-of-course test, REL Appalachia researchers created z-scores by subtracting the mean score for that test from each 
student’s test score and dividing it by the standard deviation. Researchers commonly use this approach to put different tests on a 
comparable metric or scale. See, for example, May, Perez-Johnson, Haimson, Sattar, and Gleason (2009) and Clark, Isenberg, Liu, 
Makowsky, and Zukiewicz (2017). 

12 In accordance with IES guidance, REL Appalachia researchers did not conduct statistical significance testing because the study data 
cover the entire population of interest. Statistical tests are typically meant to make inferences about a population when the data cover 
only a sample from that population. When data come from a census, there is no statistical uncertainty in descriptive findings (with some 
exceptions, such as when data are missing). Differences between groups in the data are, in fact, the true differences in the population. 

13 For continuous outcomes such as test scores, the WWC has adopted the most commonly used effect size index, the standardized mean 
difference known as Hedges' g. The WWC's effect size measure of choice for dichotomous outcomes is the Cox index, which yields effect 
size values similar to the values of Hedges' g. Both effect size indices increase magnitude as the difference between the groups on the 
outcome measure get larger; however, Cox's index also increases as proportions approach O or 1. 


Exhibit 2. Students who enrolled in Virtual Virginia online courses represented a small fraction of students 
who completed the associated end-of-course exams virtually or face-to-face, 2015/16-2016/17 


Outcome and subject area Face-to-face population Virtual Virginia population 
Core academic courses 927,811 426 
Mathematics SOL 417,792 201 
English language arts SOL 381,970 154 
Science SOL 128,049 71 
AP courses 214,289 3,377 
AP languages 3,417 136 
AP history/social science 106,974 2,143 
AP science 27,859 437 
AP English language arts 41,835 291 
AP mathematics 34,204 370 


Note: AP is Advanced Placement. SOL is Standards of Learning. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Students who took Virtual Virginia online courses and completed associated end-of-course exams 
were demographically different from their face-to-face course peers. 


Students in VVA courses who completed SOL or AP exams were more likely to be White and less likely to be Black 
or Hispanic than their peers in face-to-face courses (exhibits 3 and 4); they were also more likely to be gifted. 
Students who took core academic courses through VVA and completed SOL exams were also less likely than their 
face-to-face course peers to be economically disadvantaged or English learner students (exhibit 3). The 
demographic differences observed may be due to unknown factors influencing students’ decisions to take a course 
online versus face to face, to take the associated end-of-course exam, or both. Racial/ethnic minority students, 
economically disadvantaged students, students with a disability, and English learner students constituted a small 
portion of the population for both online and face-to-face AP courses (exhibit 4). 


Relative to students in face-to-face courses, more of the students who enrolled in and completed an 
end-of-course exam through Virtual Virginia resided in rural areas, particularly for Advanced 
Placement courses. 

Some 39 percent of the VVA students who enrolled in and completed an exam in core academic courses resided 
in arural locale, compared with 28 percent of face-to-face course students in the study population (exhibit 3). For 
AP courses, students residing in rural areas represented nearly half the VVA enrollment (46 percent), compared 
with 20 percent of face-to-face course students in the study population (exhibit 4). For context, the percentage of 
Virginia students in rural public elementary and secondary schools during the 2015/16 school year was 25 
percent.” The study findings are consistent with one of Virginia’s virtual learning goals: to serve rural students for 
whom courses may be unavailable due to a lack of highly qualified instructors or too few students to offer the 
course. 


4 Source: https://nces.ed.gov/pubs2018/2018052/tables/table_04.asp 


Exhibit 3. Students who enrolled in a core academic course and completed an end-of-course Standards of 
Learning exam through Virtual Virginia were demographically different from those in the same face-to-face 
courses, 2015/16-2016/17 


Substantive 


Face-to-face courses . 
difference 


Black or African 
American 


Hispanic or Latino 


Economically 
disadvantaged 


Student with a 
disability 


English learner 
student 


o 08 08/0/90 0 


Gifted 


Rural 


Notes: N = 927,211 face-to-face course students with end-of-course SOL exam scores, and N = 404 VVA online students with end-of-course SOL exam scores. 
Hispanic or Latino is defined as a person of Cuban, Mexican, Puerto Rican, South American, Central American or other Spanish culture or origin, regardless 
of race. Black or African American is defined as a person having origins in any of the black racial groups of Africa. The bars represent the percent of students 
in each demographic group, for VVA and face-to-face courses. The denominator of each bar is the total number of students who took the SOL exam and 
enrolled in the course in VVA or face-to-face format. For example, 67 percent of students enrolled in a VVA core course, who also took the SOL for that 
course, were White. 

* Substantively important difference where the effect size (Cox’s index) is equal to or greater than 0.25. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit 4. Students who enrolled in an Advanced Placement course and completed an Advanced Placement 
exam through Virtual Virginia were demographically different from those in the same face-to-face courses, 
2015/16-2016/17 


Substantive 
VVA online AP courses ia Face-to-face AP courses 


difference 
White 2 


6%* 
Black or African 4 : 
American = 9% 


Hispanic or Latino 
a» 
Economically aor 14% 


disadvantaged 


Student with a 
disability 


English learner 
student ) 


Gifted 


Rural 


rary 
e,;o0 9 8;8 8; @ 


20% 


Notes: N = 214,435 face-to-face course students with AP exam scores, and N = 3,396 Virtual Virginia online students with AP exam scores. Hispanic or Latino 
is defined as a person of Cuban, Mexican, Puerto Rican, South American, Central American or other Spanish culture or origin, regardless of race. Black or 
African American is defined as a person having origins in any of the black racial groups of Africa. The bars represent the percent of students in each 
demographic group, for VVA and face-to-face courses. The denominator of each bar is the total number of students who took the AP exam and enrolled in 
the course in VVA or face-to-face format. For example, 72 percent of students enrolled in a VVA AP course, who also took the AP exam for that course, were 
White. 

* Substantively important difference where the effect size (Cox’s index) is equal to or greater than 0.25. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


There was some geographic clustering of students in the Virtual Virginia population in more 
metropolitan areas in parts of the state and in some more rural areas as well. 

Students who enrolled in and completed end-of-course exams in VVA were somewhat geographically clustered 
(exhibit 5). Several of the school divisions with higher numbers of students in the VVA population were located in 
more metropolitan areas of northern Virginia (Prince William County and Stafford County) and southeast Virginia 
(Virginia Beach City, Chesapeake County, Hampton City, York County, and Suffolk City). However, there were also 
several high-use divisions in more rural areas of Virginia (such as Tazewell, Pittsylvania, Patrick, Accomack, and 
Augusta counties). 


Students who enrolled in a Virtual Virginia course and completed the end-of-course exam did so later 
in their academic trajectory on average than their face-to-face course counterparts. 

Except for English language arts (English 11 SOL), students enrolled in VVA courses later in their academic 
trajectory than their face-to-face course counterparts (exhibit 6). For example, only 24 percent of VVA students 
with end-of-course exam scores in core science courses were in grade 9, compared with 42 percent of face-to- 
face core science course students. Instead, more of these VVA students were in grade 10 (44 percent versus 38 
percent of face-to-face course students). Whereas very few students took the face-to-face course in grade 12 (4 
percent), 13 percent of the VVA students were in grade 12. Although it could not be determined reliably from the 
available data, some students may take VVA courses after failing the course face to face in a prior grade. Providing 
credit recovery opportunities is one of the most common purposes for offering online courses in K-12 educational 
settings (Clements, Stafford, Pazzaglia, & Jacobs, 2015; Heppen et al., 2016b; Queen & Lewis, 2011) and may be 
among Virginia school and division leaders’ reasons for offering VVA courses to students. 
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Exhibit 5. There was some geographic clustering of students who enrolled in a Virtual Virginia course and completed the end-of-course exam, 
2015/16-2016/17 


Virtual Virginia Overall Population 


fs Oo 0 

y O 1-10 
‘i Gi 11-50 
i 51-100 
i 101-200 


Virtual Virginia Core Academic Course Population Virtual Virginia AP Course Population 
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O 110 
Gi 11-50 
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Note: The map represents the number of students who both enrolled in Virtual Virginia (VVA) online courses and completed the end of course exam. Virginia school divisions with higher numbers of students in 
the VVA population are represented by darker shades of blue, while those with lower numbers of students are represented by lighter shades of blue. N =401 VVA online students with end-of-course SOL exam 
scores and N = 3,377 VVA online students with end-of-course AP exam scores. Twenty-five VVA online students with end-of-course SOL exam scores did not have school division information and are not included 
in the map. 

Source: Author’s analysis of administrative data provided by the Virginia Longitudinal Data System, 2015/16 and 2016/17. 


Exhibit 6. Students who enrolled in Virtual Virginia online core academic courses and completed end-of- 
course Standards of Learning exams took the course in later grades in their academic trajectory on average 
than their face-to-face course counterparts, 2015/16-2016/17 


[2 Students enrolled in VVA online courses Students enrolled in face-to-face courses 
Overall Mathematics 
40% 
35% : 35% 
said "30% «= 32% __ i 32% >” 30% 

23% 26% : 22% 

L 17% 14% 

99 I 
Li ar ~ ae ti 5% 4% 
a al = 
Grade 9 Grade 10 Grade 11 Grade 12 I Grade 8 Grade9 Grade10 Grade11 Grade 12 
English language arts Science 
89% 92% 
a4 
42% 320% 
24% 
| 19% 
: | °16% = 13% 

— _ = 
Grade 9 Grade 10 Grade 11 Grade 12 } Grade 9 Grade 10 Grade 11 Grade 12 


Note: Ns vary by subject. N = 927,815 face-to-face course students overall with end-of-course SOL exam scores, and N = 430 Virtual Virginia online students 
overall with end-of-course SOL exam scores. The bars represent the percent of students in each grade, for VVA and face-to-face courses. The denominator 
of each bar is the total number of students who took the SOL exam and enrolled in the course in VVA or face-to-face format. For example, 23 percent of 
students enrolled in a VVA core course, who also took the SOL for that course, were in grade 9. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


For AP courses, grade 12 students constituted more of the VVA population than the face-to-face course population 
(exhibit 7). For example, 55 percent of face-to-face course students with AP science exam scores were in grades 
9, 10, or 11, compared with 38 percent of VVA students with AP exam scores. Nearly two-thirds (61 percent) of 
the VVA students were taking the AP science course in grade 12, whereas fewer than half (44 percent) of the face- 
to-face course students were. 
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Exhibit 7. Students who enrolled in Virtual Virginia online Advanced Placement courses and completed 
Advanced Placement exams took the course in later grades in their academic trajectory on average than their 
face-to-face course counterparts, 2015/16-2016/17 
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AP overall AP mathematics 
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: 
Note: Ns vary by subject. N = 214,435 face-to-face course students overall with AP exam scores, and N = 3,396 Virtual Virginia online students overall with 
AP exam scores. The bars represent the percent of students in each grade, for VVA and face-to-face courses. The denominator of each bar is the total number 
of students who took the AP exam and enrolled in the course in VVA or face-to-face format. For example, 23 percent of students enrolled in a VVA AP course, 
who also took the AP exam for that course, were in grade 9. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


On average, students in the Virtual Virginia population earned higher scores on prior measures of 
their academic achievement than their face-to-face course peers, except for core mathematics 
courses. 


On average, VVA students taking core academic courses and completing end-of-course exams scored higher on 
the English language arts and science SOL exams they had taken in grade 8 than their face-to-face course peers 
but lower on mathematics SOL exams (exhibit 8). VVA students also took lower-level mathematics courses in grade 
8 than their face-to-face course peers, with 83 percent of VVA students taking grade 8 mathematics, compared 
with only 56 percent of face-to-face course students (data not shown). Only 13 percent of students in the VVA 
population took algebra | in grade 8, compared with 29 percent of students in the face-to-face population (data 
not shown). For AP courses, VVA students scored higher on average than their counterparts on their past SOL 
exams in math, as well as English language arts and science (exhibit 8). 
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Exhibit 8. Students who enrolled in courses through Virtual Virginia and completed the associated end-of- 
course exams typically performed higher on the Standards of Learning exams they took before entering high 
school than their face-to-face course peers, 2011/12—2015/16 


Prior mathematics achievement 


is VVA online courses 


aa Face-to-face courses 


515* 
SOL passing SOL passing 
score (400) score (400) 
668 academic course papiiation: — AP course spapulaton 
Prior English language arts achievement 
511* 
445* 
SOL passing 
SOL passing score (400) 
score (400) 
Gore seadantie aeademie course population AP course pepillaton 
Prior science achievement 
524* 
SOL passing ‘SOL passing 
score (400) score (400) 
Core academic course population AP course population 


Note: Ns vary by subject. Students’ prior achievement was measured by their scores on the grade 8 reading SOL for English language arts, the grade 8 science 
SOL for science, and the mathematics SOL the student took when he or she was in grade 8 (such as grade 8 math or algebra | end-of-course SOL). The prior 
achievement exam in mathematics varied because of variation in course-taking pathways. 

* Substantively important difference where the effect size is equal to or greater than 0.25. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


With the exception of English language arts courses, Virtual Virginia students generally scored lower 
on end-of-course Standards of Learning and Advanced Placement exams than similar students taking 
the course face to face. 


Students taking VVA courses scored lower on end-of-course SOL and AP exams on average than their face-to-face 
course counterparts in mathematics and science, after the study team applied statistical controls to account for 
student demographic characteristics and their prior achievement. The study team observed a similar pattern for 
AP history courses. These differences were substantively important (exhibit 9). For English language arts, however, 
students in VVA and face-to-face courses performed similarly on both SOL and AP end-of-course exams. 


The size of the difference in average achievement between the online and face-to-face course populations can be 
quantified in terms of the effect size? and change in percentile rank (exhibit 9). For example, an estimated effect 
size of —-0.24 for AP science means that, holding other observed variables constant, a student in the face-to-face 
course who scored at the 50th percentile on the exam would be predicted to score at the 41st percentile had the 
student taken the course through VVA (a change in percentile rank of —9). 


Exhibit 9. Exam scores were lower in Virtual Virginia courses than in face-to-face courses for most subjects, 
2015/16-2016/17 


Outcome and subject area Estimated effect size Boenase:in percentile tank trom tace-ta-tare 


course to VVA course 


Core academic courses 


Mathematics SOL -0.40* -16 
English language arts SOL -0.05 -2 
Science SOL -0.32* -13 
AP courses 

AP languages -0.19 -8 
AP history/social science -0.31* -12 
AP science -0.24 -9 
AP English language arts -0.06 -2 
AP mathematics -0.35* -14 


Note: The expected change in percentile rank for face-to-face course student who scored at the 50th percentile on the exam had the student taken the 
course through Virtual Virginia. 

* Substantively important difference where the effect size is equal to or greater than 0.25. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


With the exception of English language arts courses, the likelihood of scoring proficient on a 
Standards of Learning or Advanced Placement exam was lower for students taking the course 
through Virtual Virginia than for those taking the course face to face. 

Students taking VVA courses and completing end-of-course exams appeared to be less likely to earn proficient 
scores on mathematics and science exams than students taking the course face to face, again after controlling for 
available student characteristics (exhibit 10). For core academic courses, the difference was largest for 
mathematics (30 percentage points), followed by science (17 percentage points). No substantively important 
differences were observed between online and face-to-face course outcomes for English language arts courses. 


15 An effect size is a standardized measure of the magnitude of an effect and is calculated as the standardized mean difference between 
two groups. 
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Among AP courses, the difference was largest for AP mathematics (19 percentage points), followed by AP history 
(18 percentage points), AP science (16 percentage points), and AP languages (5 percentage points). No 
substantively important differences were observed between online and face-to-face course outcomes for AP 
English language arts. 


Exhibit 10. For most subject areas, the likelihood of attaining a score of proficient or above on the Standards 
of Learning or Advanced Placement exam was lower for students in Virtual Virginia courses than for students 
in face-to-face courses, 2015/16—2016/17 


LL] VVA online courses Face-to-face courses 


SOL AP 
86% 88% aia 87% ge%r 22% 
‘0 
70%* 68% 68% 64% 68% 
54% 
51%* Ais ” 50%* 
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ELA Math Science ELA Math Science History Language 


Note: The bars represent the predicted percentage chance of attaining a score of 400 or above on an SOL exam or 3 or above on an AP exam, holding 
constant other factors, such as demographics and prior achievement. In other words, it shows how likely a VVA student is to earn a passing score on the SOL 
or AP exam compared with a student in a face-to-face course, assuming they have similar background characteristics. 

* Substantively important difference where the effect size (Cox’s index) is equal to or greater than 0.25. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Limitations 


Because the study design was correlational, it is difficult to determine if the differences between online and face- 
to-face performance are caused by student characteristics, VVA program characteristics, or some other factor 
such as students’ reasons for enrolling in the course or the supports available in their local schools. More evidence 
and more rigorous and controlled studies, coupled with qualitative studies of VVA implementation, will be needed 
to further interpret the preliminary findings in this study. 


The data used to conduct the analyses were not representative of all VVA enrollments. Not all students who enroll 
in courses complete them, and online courses often have a higher noncompletion rate than face-to-face courses 
(Bawa, 2016; Freidhoff, 2016). Additionally, end-of-course SOL exams in Virginia do not cover all courses and 
subjects (including all VVA courses), and not all students enrolled in courses with end-of-course SOL or AP exams 
take the exams. In this study, just 48 percent of students enrolling in VVA courses completed the end-of-course 
exam, compared with 81 percent of students in face-to-face courses. The available administrative data on the 
reasons for course “withdrawals” prior to test completion are very limited, and the information that is recorded 
is too unreliable to support a meaningful comparison of students who enroll in VVA but do not complete with 
those who do. For example, in some cases schools upload an entire roster of students into VVA when they have 
no face-to-face teacher for the course, and once a teacher is hired, they “withdraw” the students and place them 
in a face-to-face course. 
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The population of VVA students was small, particularly for core academic courses, so researchers pooled 
enrollments into subject areas to increase the available number of online course enrollees. This approach could 
have masked important differences in outcomes by course and for student subpopulations. Additional research is 
needed to look more closely at performance among subgroups of students such as economically disadvantaged 
students, students living in rural areas, English learner students, students taking the course for credit recovery, 
and students enrolled in VVA’s full-time enrollment model. 


Lastly—and importantly—VDOE and VVA do not prescribe specifically how students, schools, or school divisions 
should use VVA. Thus, there is considerable variation in the ways the program is used across the state. For 
example, 51 percent of all schools with students enrolled in VVA courses also had students enrolled in the same 
course in a face-to-face format.’® The reasons students are taking online courses, the kinds of students 
encouraged to take online courses, and the local supports available to students may contribute to the differences 
in exam scores (Freidhoff, 2016; Taylor et al., 2016). Such factors could not be measured with the available data. 
Similarly, this study was not able to address course rigor or instructional quality or the comparability of these 
across VVA and face-to-face settings. Furthermore, these results may not generalize to online settings other than 
VVA. 


Implications 


These analyses demonstrate that students enrolled in core academic and AP online courses through VVA differ, 
on average, from students in face-to-face courses on measurable characteristics (particularly prior academic 
achievement) and are also likely to differ on unobserved factors (such as school engagement or academic 
motivation) that could contribute to differences in their performance on end-of-course exams. 


Despite the limitations above, these initial analyses offer some learning and suggestions for next steps. First, this 
study raised a research question regarding students’ academic achievement in VVA courses that could be 
addressed empirically; documented procedures for processing, cleaning, analyzing, and reporting data in the VLDS 
to answer the question; and established a methodology that VDOE can replicate to monitor whether the 
correlational findings hold or change for successive cohorts of students over time. Replicating the regression 
analyses may be particularly informative if a randomized controlled study of VVA is not feasible. This study may 
also serve as a roadmap for other states interested in comparing student performance in online and face-to-face 
courses using data from statewide longitudinal data systems. 


Second, the current study provides a rich portrait of who enrolls in VVA online courses and how these students 
are similar to and different from their peers who take and complete the same course in a face-to-face setting. But 
the reasons for the observed differences, and how these are related to differences in academic outcomes, remain 
unknown. Getting a better understanding of why and how students are identified for VVA courses, and the factors 
that are related to both course completion and end-of-course test taking, such as local supports, would be an 
important next step for state education leaders to consider. This could include, for example, asking school 
counselors to report students’ reasons for enrolling on the VVA course registration form or surveying students 
dropping out of VVA courses to understand their reasons for withdrawing. Additional data on selection of and 
retention in online courses may support more rigorous quasi-experimental study designs. They would also provide 


16 The VLDS does not capture data on which courses were offered in a given school, only on which courses had students enrolled. Given 
that courses may be cancelled if take-up is low, or only offered when a particular need arises (for example, a temporarily home-bound 
student), the difference between offerings and enrollments may be substantial. As such, it is not appropriate to use enrollments as a 
proxy for offerings. 


18 


an opportunity to compare the characteristics of students who enroll in VVA but do not complete with those who 
do complete. 


Lastly, this study focused on only one short-term academic outcome: performance on end-of-course exams. 
Different relationships may be observed between taking a course online through VVA and other short-term and 
longer-term outcomes, such as subsequent course-taking patterns and performance, staying on track 
academically and graduating, or enrolling in college. Future research could also examine whether VVA is an 
effective way to broaden access to courses that are not available to students at their physical school. Offering 
online courses in settings where the course is otherwise not available could still be beneficial to the students who 
take advantage of them, as Heppen et al. (2011) found, even if they underperform relative to students in a face- 
to-face course. 
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At SRI International 


Appendices: Comparing the achievement of students in 


Virtual Virginia and face-to-face courses 


The Virginia Department of Education’s Virtual Virginia (VVA) program delivers more than 90 online courses to 
more than 10,000 middle and high school students each year. The department partnered with Regional 
Educational Laboratory Appalachia to examine the relationship between students’ participation in a subset of 31 
VVA online courses and their performance on end-of-course Virginia Standards of Learning and Advanced 
Placement exams during the 2015/16 and 2016/17 school years. These are appendixes to the report Comparing 
the achievement of students in Virtual Virginia and face-to-face courses (Mislevy, Schmidt, Puma, Ezekoye, & 
Saucedo 2020). They provide details on the data sources, study methodology, and results of sensitivity analyses. 


Appendix A. Outcomes, data sources, and methods 


Appendix B. Detailed analysis exhibits 


Appendix C. Sensitivity analyses 


Appendix A: Outcomes, data sources, and methods 


This appendix describes in detail the outcome measures used in the analysis, data sources and key terms, data 
intake and cleaning procedures, and key analysis decisions. 


Outcome measures 


Given the importance of having comparable outcomes across online and face-to-face courses, the academic 
achievement outcome of interest was performance on standardized exams, namely the state of Virginia’s 
Standards of Learning (SOL) end-of-course tests and the College Board’s Advanced Placement (AP) exams. 


Standards of Learning exams 


The SOL establish minimum expectations for what Virginia public school students should know and be able to do 
at the end of each grade or course in English, mathematics, science, history/social science, and other subjects.” 
SOL exams in reading, writing, mathematics, science, and history/social science measure students’ success of in 
meeting the Board of Education’s expectations for learning and achievement. Virginia classroom teachers review 
all items on SOL exams for accuracy and fairness, and teachers also assist the state Board of Education in setting 
proficiency standards for the tests. A list of all the SOL exams administered in grades 8 through 12 is in exhibit A1. 


V7 Source: http://www.doe.virginia.gov/testing/index.shtml 


Exhibit A1. Standards of Learning (SOL) exams by grade level 


Grade level 
High 


SOL content area Grade 8 Yoel afeye)| 
English: reading e e 


English: writing 
Mathematics 
Science 

Algebra | 
Geometry 
Algebra Il 
Virginia and U.S. history 
World history | 
World history II 
World geography 
Earth science 
Biology 
Chemistry 


Source: Virginia Department of Education 


SOL exams in English reading, mathematics, science, and history/social science have 35 to 50 items or questions 
that measure content knowledge, scientific and mathematical processes, reasoning, and critical thinking skills. 
English writing skills are measured with a two-part exam that has multiple-choice items and an essay. Student 
performance is graded on a scale of 0 to 600, with 400 representing the minimum level of acceptable proficiency 
and 500 representing advanced proficiency. The grades 3—8 SOL exams are vertically aligned, while the end-of- 
course exams (such as the algebra | SOL and geometry SOL) are not.*® On English reading and mathematics tests, 
the Board of Education has defined three levels of student achievement—basic, proficient, and advanced—with 
basic describing progress toward proficiency. 


Advanced Placement exams 


High schools can choose to offer AP courses developed by the College Board’s AP program to allow students to 
pursue college-level course work in high school. Each course has a corresponding exam, and passing the exam 
enables high school students to earn college credit for courses they take in high school. 


The exams are scored on a scale of 1 to 5. A score of 5 indicates the student is extremely well qualified to receive 
college credit for that course. A score of 4 means the student is well qualified, and a score of 3 indicates the 
student is qualified. A score of 2 means the student is possibly qualified, and a score of 1 offers no 
recommendation for college credit.*? 


Colleges decide what constitutes a passing score on an AP exam, and this can vary depending on the exam's subject 
matter. Colleges generally give credit for a score of 3 or higher, although more competitive colleges may give 


18 Source: http://www.doe.virginia.gov/boe/accreditation/2017-soa-impact.shtml 
19 Source: https://apstudents.collegeboard.org/about-ap-scores/ap-score-scale-table 


credit only for a 4 or higher, depending on the exam.” For the purposes of this study, REL Appalachia researchers 
defined achieving proficiency as earning a score of 3 or higher on an AP exam. 


In addition to AP and SOL exam scores, the REL Appalachia study team explored several other academic 
outcomes with Virginia Department of Education (VDOE) staff and ultimately excluded them from the study 
because the data were not collected (as was the case for credit attainment), not available within the study time 
frame (for example, subsequent course enrollment), or not measured consistently across schools/divisions or 
course format (for example, final grades, incompletes, withdrawals). This is discussed further under “Data intake 
and cleaning.” 


Data sources and key terms 


REL Appalachia researchers requested student-level data from VDOE via the Virginia Longitudinal Data System 
(VLDS).?2 Researchers received 14 data files with more than 40 million records across six years. VDOE also provided 
demographic and academic information on each student, including scores on statewide assessments and AP 
exams. The specific data that researchers obtained from the VLDS data tables are listed in exhibit A2. 


Exhibit A2. Data sources and variables used in analysis 


VLDS table Filters Data elements used 
Student records 2015 and 2016 Student demographics 
Grades 8-12 Grade level 
School 
School division 
Student schedules 2015 and 2016 Face-to-face course enrollment 
Grades 8-12 
State assessment results 2011 through 2016 SOL exam scores (outcome and baseline) 
Grades 7-12 
AP courses Grades 9-12 AP exam scores (outcome only) 
Virtual Virginia None Virtual Virginia course enrollment 


Source: Virginia Longitudinal Data System (VLDS) 


The analysis also included two variables that were not available from VLDS: locale code and percentage rural. 
Using the student’s school of record from the VLDS, researchers linked the locale classification from the National 
Center for Education Statistics Common Core of Data and the percentage of the region in which the school was 
located that was rural from the United States Census. 


Definitions for the key terms and variables used in this analysis are in exhibit A3. 


Exhibit A3. Definitions of variables and key terms 
Term DY=yitavidieyay 


City Comprises three locale codes for territories inside an urbanized area and inside a principal city, as follows: 
(11) City—Large: 250,000 or more 
(12) City—Midsize: Less than 250,000 and greater than or equal to 100,000 
(13) City-Small: Less than 100,000 


Division name The name of the locality of the school division 


Division number Three-digit state-assigned number for a division that reported the student data to the VDOE 


20 Source: https://apstudents.collegeboard.org/about-ap-scores 


21 Source: http://vids.virginia.gov/ 


Economically Identifies a student as economically disadvantaged if he or she meets any one of the following: is eligible 
disadvantaged for free or reduced-price meals, receives Temporary Assistance for Needy Families, is eligible for Medicaid, 
or identifies as either migrant or experiencing homelessness 


English learner Identifies students who are between 3 and 21 years old, enrolled or preparing to enroll in an elementary or 
secondary school, and are one of following: 
e Not born in the United states or whose native languages are not English 
e = Are Native American or Alaska Native, and come from an environment where languages other than 
English have a significant impact on their level of language proficiency 
e Migratory, whose native languages are not English, and the dominant languages in the environment 
where they come from are not English 
The students must additionally have difficulties speaking, reading, writing, or understanding the English 
language, which prevent the students’ ability to meet the state's proficient level of achievement on state 
assessments, successfully achieve in classrooms where the language of instruction is English, or participate 
fully in society 


Gender Identifies the student’s gender 


Gifted Those students in public elementary, middle, and secondary schools beginning with kindergarten through 
grade 12 who demonstrate high levels of accomplishment or who show the potential for higher levels of 
accomplishment when compared with others of the same age, experience, or environment. Their aptitudes 
and potential for accomplishment are so outstanding that they require special programs to meet their 
education needs. These students will be identified by professionally qualified individuals through the use of 
multiple criteria as having potential or demonstrated aptitudes in one or more of the following areas: 
general intellectual aptitude; specific academic aptitude; career and technical aptitude; or visual or 
performing arts aptitude. 


Grade The grade level at which the student receives services in a school or an education institution during a given 
academic session 


Receives special Identifies individuals having a disability and who, by reason thereof, receive special education and related 
education services — services under the Individuals with Disabilities Education Act (IDEA) according to an individualized 
education program (IEP), individual family service plan (IFSP), or service plan 


Rural Comprises three locale codes for census-defined rural territories: 
(41) Rural-Fringe: less than or equal to 5 miles from an urbanized area, as well as rural territory that is less 
than or equal to 2.5 miles from an urban cluster 
(42) Rural—Distant: more than 5 miles but less than or equal to 25 miles from an urbanized area, as well as 
rural territory that is more than 2.5 miles but less than or equal to 10 miles from an urban cluster 
(43) Rural-Remote: more than 25 miles from an urbanized area and also more than 10 miles from an urban 


cluster 
School name The name of the school 
School number Four-digit state-assigned number for a school within the school division 
Subject The academic content or subject area (for example, arts, mathematics, reading, or a foreign language) 


being evaluated 


Suburban Comprises three locale codes for territories outside a principal city and inside an urbanized area: 
(21) Suburban—Large: 250,000 or more 
(22) Suburban—Midsize: less than 250,000 and greater than or equal to 100,000 
(23) Suburban—Small: less than 100,000 


Town Comprises three locale codes for territories inside an urban cluster: 
(31) Town-Fringe: less than or equal to 10 miles from an urbanized area 
(32) Town—Distant: more than 10 miles and less than or equal to 35 miles from an urbanized area 
(33) Town—Remote: more than 35 miles from an urbanized area 


Source: Virginia Longitudinal Data System (VLDS); National Center for Education Statistics. (2019b). NCES locale codes. Retrieved from 
https://nces.ed.gov/programs/handbook/data/pdf/appendix_d.pdf 
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Data intake and cleaning 


This section details the process used to clean and arrange the data for analysis. It is intended to provide enough 
detail that VDOE or future researchers could replicate the process and produce the same results. 


The cleaning proceeded in four stages: creating a demographics dataset for the Virginia student population, 
creating a course data file, creating an SOL exam scores file, and creating an AP exam scores file. 


Creating the demographics dataset 

To create the demographics dataset, the team first narrowed the student files to one record per student per year 
in the outcome years (2015/16 and 2016/17). This entailed four steps: creating analysis variables, dropping 
records that were duplicates on all analysis and other key variables, dropping students who were not attending 
Virginia schools at the time of the outcome, and addressing duplicate records in which students appeared in the 
same year with conflicting values for race, gender, and other demographic variables. 


First, the team created a set of analysis variables. The team created a continuous age variable that censored 
outliers, replacing all ages greater than 22 with the value of 22 and all ages less than 12 with the value of 12 (674 
of 3 million records). The team also dichotomized gifted codes into gifted/nongifted and disability types into 
disability/no disability. Next, the researchers reduced the dataset to the demographic variables used in analysis 
and dropped any records that were duplicates on all variables, reducing the dataset to 1.3 million records. Inactive 
students were dropped (that is, those who were previously enrolled but were no longer attending Virginia schools; 
164,625 records). Finally, the study team addressed duplicates in which students appeared more than one time 
in the same year with conflicting values for demographic variables. The prevalence of this issue was usually very 
low (less than 1 percent of records), except for gifted status (15.6 percent of records) and economically 
disadvantaged status (5.5 percent of records) (exhibit A4). 


Exhibit A4. Students appearing more than one time in the same year with conflicting demographic variables 


Only one value per year Two or more different values in the same year 


Age 986,460 357 
100% 0.04% 

Hispanic 986,077 740 
100% 0.07% 

Economically disadvantaged status 932,956 53,861 
95% 5.5% 

Gifted status 833,268 153,549 
84% 15.6% 

English learner 984,785 2,032 
100% 0.2% 

Receives special education services 974,299 12,518 
99% 1.3% 

Male 986,647 170 
100% 0.02% 

Race 985,288 1,529 
100% 0.2% 

School number 967,505 19,312 
98% 2.0% 

Grade level 976,846 9,971 
99% 1.0% 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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To address the duplicate records with conflicting demographic values, the study team used three approaches. For 
age, the team created an average-age variable for the 0.04 percent of students who had more than one value for 
age in the same school year; for sensitivity analyses, the team also kept the maximum and minimum age recorded 
for the student in the year. For race, ethnicity, economically disadvantaged status, gifted status, English learner 
status, disability status, and gender (male), the team created a variable that reflected whether the student ever 
appeared with a “yes” for the variable (if the student was recorded as economically disadvantaged in any record 
in the year, he or she was recorded as economically disadvantaged for the entire year); again, for sensitivity 
analyses, the team also kept the inverse of this variable (for example, if a student was ever recorded as not 
economically disadvantaged in any record in the year, he or she was recorded as not economically disadvantaged 
for the entire year). Finally, for student race, division, school, and grade level, the team selected the earliest 
instance in the school year; for sensitivity analyses, the team also kept all later versions. 


This first stage of data cleaning returned a Virginia student population dataset of 986,830 students (exhibit A5). 


Exhibit A5. Virginia population of students in target study grades, by year and grade level 


Grade 2015 2016 Total 
8 97,173 97,164 194,337 
9 107,568 107,720 215,288 
10 101,657 101,670 203,327 
11 93,114 95,788 188,902 
12 92,594 92,382 184,976 
Total 492,106 494,724 986,830 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Creating a course data file 


In the second stage, the team created a course data file by combining Virtual Virginia (VVA) and face-to-face 
courses and removing duplicates in which students appeared more than once in the same course in the same 
year. The team completed four steps: cleaning face-to-face course files, merging face-to-face courses with the 
Virginia student population demographics dataset, reading in VVA course files, merging VVA courses with the 
demographics dataset, and appending the face-to-face and VVA course files. 


First, the team cleaned data files containing the course records for grades 8-12 in 2015/16 and 2016/17. After 
removing duplicates on all variables and reshaping to include one record per student per course per year, the file 
included approximately 9.5 million records. Next, the team investigated the “credit awarded flag” and “final 
grade” variables, to determine whether they could be used to indicate whether students had completed the 
course successfully. Based on prior discussions with VDOE, the team expected that the data quality in these 
variables would not be sufficient to use them as outcomes. As expected, the credit awarded flag was not available 
for any records in 2015/16, and 43 percent of records in 2016/17 had conflicting values for this variable in the 
same course. Likewise, 22 percent of records did not have a final grade for the course, and 10 percent of the 
records had conflicting values for the final grade. Additionally, the records that did have values for final grade 
varied greatly in the type of value recorded. For example, some records included letter grades A through F, 
whereas others had numbers between 0 and 100, and a third group included letter codes reflecting a variety of 
other outcomes (for example, Meets expectations, Not passed, Satisfactory). Therefore, in consultation with 
VDOE, the team determined that the course files could not provide a reliable measure for course completion or 
course grades. 
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Next, the study team merged the face-to-face courses with the Virginia student population demographics dataset. 
The quality of the match is shown in exhibit A6. Approximately 7 million course records matched a student in the 
demographics dataset (76 percent of courses). Another 2.3 million courses did not match to a student in the 
Virginia student population. However, the majority of these were not the focal courses. In fact, 98 percent of all 
focal courses merged to the Virginia student population data (2 million records). Finally, 10,613 students in the 
demographics dataset for the Virginia student population (0.1 percent) did not have a course listed in the course 
files. 


Exhibit A6. Quality of the match between face-to-face courses and Virginia student population dataset 


Result Number of observations Percent of observations 
Not matched overall 2,268,453 23.9 
Not matched from courses 2,257,840 23.8 
Not matched from Virginia student population 10,613 0.1 
Matched 7,225,033 76.1 
Total 9,493,486 100 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


The study team retained all records with students in both the Virginia student population and the course file—a 
total of 7.2 million records. The number of students in each focal course and school year are in exhibit A7. 


Exhibit A7. Counts of students, by face-to-face course and school year 
School Courses for the Exchange of 


Focal course Data (SCED) course code 2015/16 2016/17 
English 11 1003 73,374 74,523 
Algebra | 2052 102,401 103,383 
Algebra II 2056 64,117 67,891 
Geometry 2072 92,130 94,470 
Earth science | 3001 76,596 78,188 
Biology | 3051 103,106 102,742 
Chemistry | 3101 64,171 65,718 
AP calculus AB 2124 13,590 13,396 
AP calculus BC 2125 2,155 2,317 
AP computer science A 10157 3,363 3,652 
AP statistics 2203 9,021 9.253 
AP environmental science 3207 8,385 8,894 
AP biology 3056 8,391 9,112 
AP physics 1 3165 2,892 5,592 
AP physics 2 3166 462 900 
AP French language and culture 24114 918 807 
AP Latin 24355 519 533 
AP Spanish language and culture 24064 2,822 3,065 
AP Chinese language and culture 24414 44 143 
AP art history 5153 924 901 
AP English literature and composition 1006 14,516 13,825 
AP English language and composition 1056 2,605 0 
AP English language and composition 1005 15,702 17,928 
AP human geography 4004 6,757 7,060 
AP U.S. history 4104 17,465 18,529 
AP world history 4057 11,355 11,449 
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AP European history 4056 3,893 4,089 
AP government and politics: U.S. 4157 17,842 17,620 
AP government and politics: comparative 4158 2,794 2,774 
AP microeconomics 4203 2,874 2,390 
AP macroeconomics 4204 1,387 567 
AP psychology 4256 13,172 13,046 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


The study team then imported to Stata (a statistical programming software) files for all VVA courses in 2014/15 
and 2015/16. These totaled 31,697 records across the two years. As with the face-to-face courses, the study team 
investigated the final grade variable in the dataset and found that it was not viable as an outcome. After removing 
duplicates, this dataset contained 30,868 records. The number of students in each VVA focal course is in exhibit 
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Exhibit A8. Counts of students, by Virtual Virginia course and school year 


Focal course NVAV/: Wool U] asx=m 1B) 2015/16 Ploncyaly, 
English 11 1150 31 72 
Algebra | 3130 32 45 
Algebra II 3135 24 75 
Geometry 3143 32 90 
Earth science | 4210 40 104 
Biology | 4310 39 63 
Chemistry | 4410 21 58 
AP calculus AB 3177 68 95 
AP calculus BC 3178 136 82 
AP computer science A 3185 62 214 
AP statistics 3192 125 138 
AP environmental science 4270 350 339 
AP biology 4370 125 87 
AP physics 1 4570 59 125 
AP physics 2 4571 19 39 
AP French language and culture 5170 45 36 
AP Latin 5370 32 36 
AP Spanish language and culture 5570 62 73 
AP Chinese language and culture 5860 35 27 
AP art history 9151 141 185 
AP English literature and composition 1195 150 168 
AP English language and composition 1196 118 109 
AP human geography 2212 262 258 
AP U.S. history 2319 100 143 
AP world history 2380 122 106 
AP European history 2399 215 192 
AP government and politics: U.S. 2445 121 282 
AP government and politics: Comparative 2450 133 122 
AP microeconomics 2802 224 149 
AP macroeconomics 2803 226 149 
AP psychology 2902 678 645 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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In the fourth step, researchers merged the VVA courses with the demographics dataset for the Virginia student 
population. The quality of the merge is shown in exhibit A9. Approximately 97 percent of all VVA records matched 
to a student in the demographics dataset (30,003 of 30,868). Among students in the focal courses, 98 percent of 
all records merged. On the other hand, 97 percent of students in the Virginia student population were not enrolled 
in any VVA courses during either year. 


Exhibit A9. Quality of the match between Virtual Virginia courses and the Virginia student population 


Result Number of observations Percent of observations 
Not matched overall 962,570 97.0 
Not matched from VVA courses 865 0.1 
Not matched from Virginia student population 961,705 96.9 
Matched 30,003 3.0 
Total 992,573 100 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Finally, the study team appended the VVA course files with the face-to-face courses and investigated the 
proportion of students taking two courses in the same subject in the same year. A total of 4,787 records appeared 
twice, once in face-to-face course and once in VVA files. This amounted to 0.2 percent of all records but 46 percent 
of VVA courses. After consultation with VDOE, the study team determined that students enrolled in both course 
types should be marked as participating in VVA for the course. In this step, the study team also dropped any 
courses that were marked as “virtual” in the course file but were not in the VVA file. (Those records represented 
virtual courses from other providers.) 


The enrollment for each focal course is shown in exhibit A10. 


Exhibit A10. Proportion of students enrolled in face-to-face, Virtual Virginia, or both course types, by focal 
course 


Focal course Face to face NAVAN Both (counted as VVA) 
AP art history 978 129 100 
AP biology 16,525 86 80 
AP calculus AB 25,576 70 66 
AP calculus BC 4,393 140 50 
AP Chinese 126 45 8 
AP computer science 6,512 79 116 
AP English language 31,918 140 55 
AP English literature 26,885 89 132 
AP environmental science 15,798 233 274 
AP European history 6,753 199 121 
AP French 789 65 9 
AP government and politics: comparative 5,352 110 78 
AP government and politics: U.S. 32,185 123 148 
AP human geography 11,960 186 189 
AP Latin 508 44 17 
AP macroeconomics 1,564 142 114 
AP microeconomics 4,881 126 130 
AP physics 1 8,409 121 37 
AP physics 2 4,089 50 6 
AP psychology 24,042 522 446 
AP Spanish 2,986 91 27 
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Focal course Face to face NAVAN Both (counted as VVA) 


AP statistics 17,056 132 89 
AP U.S. history 34,167 83 99 
AP world history 22,577 95 79 
Algebra | 196,953 28 28 
Algebra II 127,794 61 34 
Biology 196,562 24 57 
Chemistry 124,440 29 36 
Earth science 144,355 32 75 
English 11 140,149 25 54 
Geometry 157,890 47 52 
Total 1,394,172 3,346 2,806 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


After narrowing the file to the focal courses, the final course file included 1.4 million course records, with 770,375 
unique students. 


Creating a file for Standards of Learning test scores 


In the third stage, the study team created a clean file of student test scores on the SOL exams. This stage 
proceeded in three steps. First, the study team imported to the statistical analysis software Stata the 2015/16 and 
2016/17 SOL scores for students in grades 8 through 12. The 2015/16 file included 1.7 million records, and the 
2016/17 file 1.6 million. However, approximately 3 percent of students appeared more than once in the same test 
in the same year, with different scores. To address these duplicates, the study team removed records of the 
student retaking the same test by ordering the records by test date and keeping the score associated with the 
earliest test date as the main variable for the analysis. The team also imported to Stata the files of test scores for 
students in grade 8 in 2011/12 through 2014/15 and cleaned and deduplicated them in the same way as the 
2015/16 and 2016/17 data files. This file included only grade 8 students because it served as the baseline measure 
for students who took focal courses in later years. 


Second, the researchers merged the SOL exam datasets with the course files. Between 64 and 78 percent of all 
courses and SOLs matched (exhibit A11). 


Exhibit A11. Quality of the match between Standards of Learning test files and course files 
SOL did not match to a course Course did not match to an SOL SOL and courses matched 


Mathematics 107,311 112,040 394,139 
17% 18% 65% 
English language arts 65,514 9,071 133,174 
32% 4% 64% 
Science 80,726 42,340 427,269 
15% 8% 78% 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


In the third step of this stage, the study team arranged the SOL datasets to associate the appropriate outcome 
and prior achievement for each focal course. This is discussed further under “Key analysis decisions.” 


Finally, researchers examined the remaining duplicates: 3.5 percent of students took more than one course in the 
same subject area in the same year. They determined that this was caused by students in block-scheduled schools 
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who could take more than one focal course in the same year but in different semesters. These duplicates were 
retained. 


Creating a clean file for Advanced Placement test scores 

In the final stage of data cleaning, the study team created a clean file of student test scores on the AP exams. This 
stage proceeded in four steps. First, the study team imported to Stata the AP scores for students in grades 9 
through 12. The full file included 269,565 records, but it was stored as a “wide” file, with 163 variables 
corresponding to each test subject date and outcome. The researchers reshaped the file to provide one line per 
student per test. Then they identified cases where students took the same AP exam more than once and retained 
only the earliest test score (to eliminate retests). After this deduplication, only tests that were associated with the 
2015/16 and 2016/17 school years were retained. The final file included 273,866 records. 


In the second step of this stage, the study team merged the AP exam data with the Virginia student population 
demographics dataset. Nearly all AP exam records (98 percent) merged to a student in the Virginia student 
population. However, approximately 76 percent of all students in the Virginia student population were not linked 
to an AP exam. 


In the third step, the researchers merged the AP file with the course data. Most AP exams (82 percent) were 
matched to student course files. However, 18 percent of all AP exams could not be linked to courses in the same 
year. After consultation with VDOE, we determined that this mismatch was legitimate because students are 
allowed to take the AP exams without enrolling in the course. These records were dropped, however, as they 
could not be linked to the independent variable of interest (VVA versus face-to-face course-taking). 


In the fourth step, the study team arranged the AP datasets to associate the appropriate outcome and prior 
achievement for each focal course (discussed further under “Key analysis decisions”). Finally, the team examined 
the proportion of students who took more than one AP course in the same subject area in the same year. Given 
the number of AP courses grouped within each area, this was common, particularly in the history and social 
science. As with the SOL outcome, the team determined that these duplicates were legitimate and retained all 
records. In the main analysis, each of these records was treated as independent. In a sensitivity analysis, we 
examined the mean score by subject area as the outcome. 


Summarizing the final study population of end-of-course exam completers 


The final study population consisted of all Virginia public school students in grades 8-12 who enrolled in focal 
courses with either end-of-course SOL or AP exams during the 2015/16 and 2016/17 school years and completed 
the exam (exhibits A12 and A13). 
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Exhibit A12. Final study population for core academic courses with end-of-course Standards of Learning exams 


Core academic courses 


Course enrollment 


Face-to-face Virtual Virginia 


Math Science ELA Math Science 
465,320 140,115 500,685 325 103 


Exam completion 


Math Science Math Science 


Face-to-face Virtual Virginia 


90% 91% 62% 69% 


Study population 


Face-to-face Virtual Virginia 


Math Science ELA Math Science 


417,792 128,049 381,970 201 71 
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Exhibit A13. Final study population for Advanced Placement courses with Advanced Placement exams 


Advanced Placement courses 


Course enrollment 


Face-to-face Virtual Virginia 


Math Science ELA History Languages Math Science ELA History Languages 
53,326 40,712 58,773 144,419 4,405 910 1,079 540 4,409 341 


Science 
Exam completion 


Face-to-face Virtual Virginia 


Math Science ELA History Languages 


Bae 689% 7194 749 79% Math Science ELA History Languages 
‘oO ‘oO ‘oO ‘oO ‘0 


41% 41% 54% 49% 40% 


Study population 


Face-to-face Virtual Virginia 


Math Science ELA History Languages Math Science ELA History Languages 
34,204 27,859 41,835 106,974 3,417 370 437 291 2,143 136 
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Key analysis decisions 


The four key analysis decisions were the choice of outcome and baseline tests to be used, the grouping of courses 
into subject area domains, the approach to imputation, and the model specifications. 


Outcome and baseline tests used in the analysis 


In consultation with the VDOE, the study team selected the outcome tests shown in exhibit A14 for the SOL 
outcomes. The researchers allowed the baseline achievement test to vary in mathematics because of variation in 
course-taking pathways and outcome grade levels. For example, if a student took algebra | in grade 8, his or her 
baseline score may be the grade 7 mathematics exam. On the other hand, if a student took algebra | in grade 9, 
his or her baseline score may be the grade 8 mathematics exam. To account for the variation in course-taking 
pathways, the models controlled for baseline exam and outcome exam, as well as for baseline school year, 
outcome year, and outcome grade level. 


Exhibit A14. Crosswalk between focal core academic courses and their Standards of Learning outcomes and 
baseline measures 


Virtual Virginia course 


Course name SCED course code Outcome test Baseline test 
English 11 1150 1003 Reading Grade 8 reading 
Algebra | 3130 2052 Algebra | Grade 7 mathematics 

2061 Grade 8 mathematics 
Geometry 3143 2072 Geometry Grade 7 mathematics 
Grade 8 mathematics 
Algebra | 
Algebra Il 3135 2056 Algebra Il Grade 7 mathematics 
Grade 8 mathematics 
Algebra | 
Geometry 
Earth science 4210 3001 Earth science Grade 8 science 
Biology 4310 3051 Biology 
Chemistry | 4410 3101 Chemistry 


Source: Virginia Department of Education 


Likewise, in consultation with VDOE, the study team selected the SOL exams shown in exhibit A15 to use as 
baseline achievement for the AP outcomes (the outcome was always the AP exam that corresponded to the 
course). As with SOLs, the mathematics baseline measure was allowed to vary depending on the student’s course- 
taking pathway. 
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Exhibit A15. Crosswalk between focal Advanced Placement courses and their Standards of Learning baseline 
measures 
SOL exam code 


Area Course name Pretest SOLexam name Pretest 
English AP English literature 108 Grade 8 reading 
AP English language 
Math AP calculus AB 127 Grade 7 math 
AP calculus BC 137 
128 Grade 8 math 


AP computer science A 
138 


120 Algebra | 
121 Geometry 
122 Algebra Il 


AP statistics 


Sciences AP environmental science 5 
AP biology 58 Grade 8 science 
AP physics 1 165 
AP physics 2 
Languages AP French 108 Grade 8 reading 
AP Latin 
AP Spanish 
AP Chinese 
History and social AP human geography 108 Grade 8 reading 
science AP U.S. history 


AP world history 

AP European history 

AP government and politics: U.S. 

AP government and politics: comparative 
AP microeconomics 

AP macroeconomics 

AP psychology 

AP art history 


Source: Virginia Department of Education 


The study team combined test scores within a subject area to examine the relationship between taking a course 
online through VVA and SOL outcomes because too few students were taking the individual VVA courses to power 
an analysis at the course level. Because the analysis combines both outcome and baseline scores across multiple 
tests, which were not vertically aligned, the team standardized the test scores by test and year. In this approach, 
the study team created z-scores by subtracting the mean from each test score and dividing it by the standard 
deviation. Researchers commonly use this approach to put different tests on a comparable metric or scale (for 
example, Clark, Isenberg, Liu, Makowsky, & Zukiewicz, 2014). Therefore, students in this analysis were compared 
with one another based on their relative position in the distribution of scores for the same test in the same year. 
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Grouping courses into subject area domains 


The 24 focal AP courses and seven focal SOL courses fall into five subject area domains: English, mathematics, 


sciences, languages, and history and social science (exhibit A16). 


Exhibit A16. Focal courses, by subject area domain 


Subject area 


AP course name 


Core academic course name 


English AP English literature English 11 
AP English language 

Math AP calculus AB Algebra | 
AP calculus BC Geometry 
AP computer science A Algebra Il 
AP statistics 

Sciences AP environmental science Earth science 
AP biology Biology 
AP physics 1 Chemistry 
AP physics 2 

Languages AP French 
AP Latin 
AP Spanish 
AP Chinese 


History and social science 


Source: Virginia Department of Education 


AP human geography 


AP U.S. history 


AP world history 


AP European history 


AP government and politics: U.S. 


AP government and politics: comparative 


AP microeconomics 


AP macroeconomics 


AP psychology 


AP art history 


Imputing for missing values 


Imputing for missing files was done separately for the AP and the SOL exam outcomes files. The SOL exam 
outcomes file had a total of 144,168 students with an SOL outcome score and no SOL baseline score. The 
incomplete records were spread across subjects, with the most missing in chemistry and the fewest missing in 


earth science (exhibit A17). 


Exhibit A17. Number and percentage of records missing baseline scores in the Standards of Learning dataset, 


by test subject 
Algebra | 19,939 (14) 
Algebra Il 13,295 (12) 
Biology 29,979 (16) 
Chemistry 27,711 (23) 
Earth science 13,715 (11) 
English 11 20,818 (16) 
Geometry 18,711 (14) 
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Total 144,168 (15) 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Demographics were also missing for some students, but the prevalence was much lower: Demographic 
information was missing for only 717 students who had SOL outcomes scores, and in all cases all demographic 
variables were missing, including the school and division. These represent the small group of students who 
appeared in course and test files but did not appear in the demographics dataset. 


The AP exam outcomes file contained a total of 27,962 students with an AP outcome score and no SOL baseline 
score, and 27,890 were missing their baseline school year (exhibit A18). No demographic variables were missing 
for students who had AP outcomes. 


Exhibit A18. Number and percentage of records missing baseline scores in the Advanced Placement dataset, 
by test subject 


Subject Number missing (percent) 


Art history 72 (11) 
Biology 2,857 (25) 
Calculus AB 1,691 (11) 
Calculus BC 351 (10) 
Chinese 25 (16) 
Computer science 573 (11) 
English language 2,721 (11) 
English literature 2,157 (12) 
Environmental science 2,927 (27) 
European history 493 (10) 
French 81 (12) 
Government and politics — comparative 379 (12) 
Government and politics — U.S. 2,739 (12) 
Human geography 1,032 (12) 
Latin 38 (10) 
Macroeconomics 176 (10) 
Microeconomics 375 (11) 
Physics 1 1,175 (19) 
Psychology 1,938 (10) 
Spanish 316 (13) 
Statistics 1,204 (11) 
U.S. history 2,806 (11) 
World history 1,764 (9) 
Total 27,890 (14) 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Imputation in each dataset proceeded in two steps. First, logical imputation was used for missing school year and 
grade level at baseline. The study team assumed an on-time grade progression and counted backward from the 
outcome year or grade level to establish the baseline year or grade level. 


In the second step, the researchers used multiple imputation to impute for the remaining missing values. The 
team used the “mi impute” command in Stata, which triggers an iterative Markov Chain Monte Carlo method to 
impute missing values (StataCorp, 2009, p. 145). Economic disadvantage, gifted status, disability status, and 
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gender (male) were imputed using logistic regression; age and baseline scores were imputed using linear 
regression; and race was imputed using multinomial regression. These missing values were imputed based on the 
following variables: outcome test score, outcome school year, outcome grade level, pretest school year, pretest 
grade level, course subject, school, division, and a flag for enrollment in a VVA course. The team used a flag for 
enrollment in a VVA course rather than imputing separately for the two groups because of the small population 
size. This approach is approved under the What Works Clearinghouse version 4.0 standards (U.S. Department of 
Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 
What Works Clearinghouse, 2017). 


The team imputed only for demographics and baseline test scores, not for outcome scores. The team made this 
choice for two reasons. First, the population of interest for the analysis was students who completed the focal 
courses and took the end-of-course exam. As discussed, it was not possible to determine course completion from 
the course files. Accordingly, the team limited the population to students who completed the course and took the 
associated exam. Second, the imputation of outcomes would have been impractical, given that the analysis 
allowed the baseline test to vary depending on student course progression in mathematics. That is, if a student 
appeared in the file with a score on the algebra | assessment in grade 8, it was impossible to know if he or she 
should have been assessed in geometry in grade 9 or 10, as students varied in the course pathways they followed. 


Unadjusted student outcomes in Virtual Virginia courses 


As a precursor to the analysis, the study team examined the unadjusted student test scores and passing rates for 
students in VVA courses by course subject area. The results of this analysis are in exhibit A19. 


Exhibit A19. Unadjusted student outcomes in Virtual Virginia courses 


Subject area Mean scaled score Percent proficient Number of enrollments 

SOL 

Math 411.6 55% 154 
Science 433.4 76% 201 
ELA 449.6 87% 71 
AP 

Math 2.9 59% 370 
Science 2.6 49% 437 
ELA 3.1 71% 291 
History and social science 3.0 61% 2,143 
Languages 3.6 83% 136 


Notes: Percent proficient is the percentage of students who scored a 400 or above on the SOL exam or a 3 or above on the AP exam. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Model specifications 

In this study, three methodological approaches were used to address the research question. First, descriptive 
Statistics were used to examine unadjusted differences in the characteristics of students taking VVA and 
completing end-of-course exams and students with exam scores in comparable face-to-face courses. The findings 
related to student demographic characteristics and prior achievement were produced from this approach. 


Second, the study team used multivariate ordinary least squares regressions to examine the relationship between 
VVA and end-of-course SOL and AP exam scores while controlling for the influence of available student 
characteristics. These regressions aggregated exam scores across courses. For example, algebra |, geometry, and 
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algebra II SOL exam scores contributed to the “mathematics SOL” outcome. Because these exams are not scored 
on the same basis, the team used z-scores to create a common outcome metric. This transformation allowed for 
comparing student test results with one another based on their relative position in the distribution of all scores 
for the same test in the same year. Unlike descriptive statistics, multivariate regressions partially account for 
preexisting differences between students by factoring out the relationship between these background 
characteristics and student achievement outcomes. The linear regressions took the following form: 


y= Bot B,VirtualVirginia + B2X + B3C + B,Yr + B,PctRural + B;M +e 


where y is the continuous z-score for the outcome in each subject area (for example, mathematics SOLs), 
VirtualVirginia is an indicator for whether the student took the course as a part of VVA (1/0), X is a vector of 
student characteristics (grade level, age, gender, race, gifted status, English learner, disability status, economically 
disadvantaged status), C is a series of indicators for each outcome course (for example, algebra | and algebra II in 
the mathematics model, with geometry as the comparison group) and baseline course (for example, mathematics 
8 and algebra | in the mathematics model, with mathematics 7 as the comparison group) included in the model, 
Yris a series of indicators for each outcome and baseline year, PctRural is a continuous variable for the percentage 
of the population in the region of the school that is designated as rural, and M is a series of indicators for which 
student-level variables were imputed using multiple imputation. The errors were clustered at the school level 
using Stata’s “cluster” command. 


Finally, the study team used multivariate logistic regression to predict whether a student reached proficiency (a 
score of 400 or above on an SOL exam or 3 or above on an AP exam). Logistic regression estimates the relationship 
between one or more variables and a binary outcome—one that has only two possible results (proficient or not 
proficient). These regressions also combine outcomes across courses, and partially account for preexisting 
differences between students. These models used the same series of control variables as in the linear regressions. 


In accordance with guidance from the Institute of Education Sciences, REL Appalachia researchers did not conduct 
statistical significance testing because the study data cover the entire population of interest. Statistical tests are 
typically meant to make inferences about a population when the data cover only a sample from that population. 
When data come from a census, there is no statistical uncertainty in descriptive findings (with some exceptions, 
such as when data are missing). Differences between groups in the data are, in fact, the true differences in the 
population. 


REL Appalachia researchers used the criterion set by the What Works Clearinghouse (WWC, 2017) to determine 
the substantive importance of the differences in outcomes or characteristics between the students in VVA and 
face-to-face courses: an effect size—a standardized measure of the magnitude of the difference—of 0.25 or 
greater. For continuous outcomes such as test scores, the WWC has adopted the most commonly used effect size 
index, the standardized mean difference known as Hedges' g. The WWC's effect size measure of choice for 
dichotomous outcomes is the Cox index, which yields effect size values similar to the values of Hedges' g. These 
effect size indices increase in magnitude as the difference between the groups on the outcome measure get larger; 
however, Cox's index also increases as proportions approach O or 1. 


A-19 


Appendix B: Detailed analysis results 

This appendix provides the full results for analyses discussed in the main report. Presented are three types of 
exhibits: descriptive exhibits (page B-1), full model results for the Standards of Learning (SOL) outcomes (page B- 
10), and full model results for the Advanced Placement (AP) outcomes (page B-16). 


Descriptive exhibits 

Information on the demographics and baseline achievement of students in the study by exam type (SOL or AP) 
and course subject area are in exhibits B1—-B10. Three types of baseline achievement are shown: z-scores, scaled 
scores, and percentage proficient or advanced. Z-scores have a mean of 0 and a standard deviation of 1. A negative 
number indicates that the student was below the average for his or her grade level and school year, whereas a 
positive number indicates that a student was above average for his or her grade level and school year. Average z- 
scores for the overall population are always zero. 


SOL scaled scores range from O to 600, where a score of 0-399 means a student did not pass the test. A score of 
400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student 
passed a test and was classified as advanced. The means of this population usually fall in the proficient range, with 
the average baseline for some AP courses in the advanced range. 


The percentage of students proficient or advanced was derived directly from the SOL scaled scores, as described 
above. The majority of students in this population scored proficient on their baseline SOL exam, with a larger 
proportion of the students with AP outcomes falling into the advanced category. 


Standards of Learning courses 


Exhibit B1. Descriptive statistics on student demographics and baseline scores, all Standards of Learning 
courses 
Students who take the SOL exam 


Characteristic Overall NAVAN Face to face 
Grade 8 7% 0% 7% 
Grade 9 33% 23% 33% 
Grade 10 30% 35% 30% 
Grade 11 26% 32% 26% 
Grade 12 4% 9% 4% 
Native American 0% 0% 0% 
Asian 6% 2% 6% 
Black or African American 22% 17% 22% 
Hispanic 14% 7% 14% 
White 53% 67% 53% 
Pacific Islander 0% 0% 0% 
Multiple races 4% 5% 4% 
Male 50% 38% 50% 
Age 15.5 15.9 15.5 
Economically disadvantaged 37% 20% 37% 
Gifted 15% 23% 15% 
English learner 7% 0% 7% 
Receives special education services 13% 15% 13% 
Region percent rural 54 47 54 


Students who take the SOL exam 


Characteristic Overall NAVAN Face to face 
Baseline SOL z-score 0.01 0.19 0.01 
Baseline SOL scaled score 436.5 443.1 436.5 
Baseline SOL percent proficient or advanced 77 85 77 
Baseline SOL percent advanced 11 16 11 
Number of students 928,237 426 927,811 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B2. Descriptive statistics on student demographics and baseline scores, Standards of Learning 


mathematics courses 


Students who take the SOL exam 


Characteristic Overall NAVAN Face to face 
Grade 8 17% 1% 17% 
Grade 9 35% 32% 35% 
Grade 10 30% 40% 30% 
Grade 11 14% 22% 14% 
Grade 12 4% 5% 4% 
Native American 0% 0% 0% 
Asian 7% 1% 7% 
Black or African American 22% 19% 22% 
Hispanic 14% 10% 14% 
White 52% 64% 52% 
Pacific Islander 0% 0% 0% 
Multiple races 5% 5% 4% 
Male 49% 41% 49% 
Age 15.5 15.6 15.2 
Economically disadvantaged 36% 20% 36% 
Gifted 15% 21% 15% 
English learner 7% 0% 7% 
Receives special education services 11% 16% 11% 
Region percent rural 54 48 54 
Baseline SOL z-score 0.01 -0.05 0.01 
Baseline SOL scaled score 441.1 434.5 441.1 
Baseline SOL percent proficient or advanced 81 86 81 
Baseline SOL percent advanced 12 13 12 
Number of students 382,124 154 381,970 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B3. Descriptive statistics on student demographics and baseline scores, Standards of Learning English 
language arts courses 
Students who take the SOL exam 


Characteristic (oW-1¢-11 WAVE Face to face 
Grade 8 0% 0% 0% 
Grade 9 0% 1% 0% 
Grade 10 4% 3% 4% 
Grade 11 92% 89% 92% 
Grade 12 4% 7% 4% 
Native American 0% 0% 0% 
Asian 5% 0% 5% 
Black or African American 25% 11% 25% 
Hispanic 13% 8% 13% 
White 52% 72% 52% 
Pacific Islander 0% 0% 0% 
Multiple races 4% 5% 4% 
Male 55% 34% 55% 
Age 15.5 16.7 16.7 
Economically disadvantaged 40% 20% 40% 
Gifted 11% 12% 11% 
English learner 8% 0% 8% 
Receives special education services 16% 20% 16% 
Region percent rural 53 42 53 
Baseline SOL z-score 0.01 0.46 0.01 
Baseline SOL scaled score 420.8 445.3 420.8 
Baseline SOL percent proficient or advanced 67 85 67 
Baseline SOL percent advanced 7 10 7 
Number of students 128,120 71 128,049 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B4. Descriptive statistics on student demographics and baseline scores, Standards of Learning science 
courses 
Students who take the SOL exam 


Characteristic Overall NAVAN Face to face 
Grade 8 0% 0% 0% 
Grade 9 42% 24% 42% 
Grade 10 38% 44% 38% 
Grade 11 16% 19% 16% 
Grade 12 4% 13% 4% 
Native American 0% 0% 0% 
Asian 6% 4% 6% 
Black or African American 22% 18% 22% 
Hispanic 13% 6% 13% 
White 53% 68% 53% 
Pacific Islander 0% 1% 0% 
Multiple races 4% 5% 4% 
Male 50% 38% 50% 
Age 15.5 15.9 15.5 
Economically disadvantaged 37% 19% 37% 
Gifted 16% 28% 16% 
English learner 7% 1% 7% 
Receives special education services 13% 12% 13% 
Region percent rural 55 48 55 
Baseline SOL z-score 0.01 0.25 0.01 
Baseline SOL scaled score 437.0 448.2 437.0 
Baseline SOL percent proficient or advanced 76 84 76 
Baseline SOL percent advanced 11 20 11 
Number of students 417,993 201 417,792 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Advanced Placement courses 


Exhibit B5. Descriptive statistics on student demographics and baseline scores, all Advanced Placement 


courses 
Students who take the AP exam 
Characteristic Overall WAVE Face to face 
Grade 9 2% 1% 2% 
Grade 10 15% 9% 15% 
Grade 11 41% 32% 41% 
Grade 12 41% 58% 41% 
Native American 0% 0% 0% 
Asian 17% 13% 17% 
Black or African American 9% 6% 9% 
Hispanic 8% 4% 8% 
White 61% 72% 61% 
Pacific Islander 0% 0% 0% 
Multiple races 5% 4% 5% 
Male 44% 44% 44% 
Age 16.8 17.0 16.8 
Economically disadvantaged 13% 14% 13% 
Gifted 42% 58% 42% 
English learner 2% 1% 2% 
Receives special education services 3% 2% 3% 
Region percent rural 52 58 52 
Baseline SOL z-score 0.10 0.42 0.10 
Baseline SOL scaled score 493.1 513.9 492.7 
Baseline SOL percent proficient or advanced 98 99 98 
Baseline SOL percent advanced 41 57 41 
Number of students 217,848 3,396 214,452 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B6. Descriptive statistics on student demographics and baseline scores, Advanced Placement 


mathematics courses 


Students who take the AP exam 


Characteristic oW-1e-11 NAVAN Face to face 
Grade 9 0% 0% 0% 
Grade 10 6% 5% 6% 
Grade 11 32% 23% 32% 
Grade 12 62% 72% 62% 
Native American 0% 0% 0% 
Asian 21% 15% 22% 
Black or African American 7% 5% 7% 
Hispanic 7% 4% 7% 
White 60% 70% 60% 
Pacific Islander 0% 1% 0% 
Multiple races 5% 6% 5% 
Male 53% 59% 53% 
Age 17.1 17.2 17.1 
Economically disadvantaged 10% 10% 10% 
Gifted 48% 67% 48% 
English learner 2% 1% 2% 
Receives special education services 3% 1% 3% 
Region percent rural 52 55 52 
Baseline SOL z-score 0.08 0.35 0.08 
Baseline SOL scaled score 495.7 515.2 495.5 
Baseline SOL percent proficient or advanced 99 99 99 
Baseline SOL percent advanced 43 58 43 
Number of students 34,574 370 34,204 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B7. Descriptive statistics on student demographics and baseline scores, Advanced Placement English 
language arts courses 
Students who take the AP exam 


Characteristic Overall WAVE Face to face 
Grade 9 0% 0% 0% 
Grade 10 0% 0% 0% 
Grade 11 57% 45% 57% 
Grade 12 42% 55% 42% 
Native American 0% 0% 0% 
Asian 15% 11% 15% 
Black or African American 10% 7% 10% 
Hispanic 7% 5% 7% 
White 62% 72% 62% 
Pacific Islander 0% 0% 0% 
Multiple races 5% 4% 5% 
Male 37% 37% 37% 
Age 17.0 17.0 17.0 
Economically disadvantaged 12% 13% 12% 
Gifted 41% 57% 41% 
English learner 1% 0% 1% 
Receives special education services 3% 1% 3% 
Region percent rural 55 59 55 
Baseline SOL z-score 0.16 0.43 0.15 
Baseline SOL scaled score 495.1 510.7 495.0 
Baseline SOL percent proficient or advanced 98 99 98 
Baseline SOL percent advanced 43 53 43 
Number of students 42,126 291 41,835 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B8. Descriptive statistics on student demographics and baseline scores, Advanced Placement science 
courses 
Students who take the AP exam 


Characteristic oW-1e-11 NAVAN Face to face 
Grade 9 3% 0% 3% 
Grade 10 9% 4% 9% 
Grade 11 43% 34% 43% 
Grade 12 45% 61% 44% 
Native American 0% 0% 0% 
Asian 19% 11% 19% 
Black or African American 9% 6% 9% 
Hispanic 7% 3% 8% 
White 59% 75% 59% 
Pacific Islander 0% 0% 0% 
Multiple races 5% 5% 5% 
Male 43% 43% 43% 
Age 16.8 17.1 16.8 
Economically disadvantaged 13% 14% 13% 
Gifted 43% 55% 43% 
English learner 2% 1% 2% 
Receives special education services 3% 4% 3% 
Region percent rural 46 57 45 
Baseline SOL z-score 0.07 0.38 0.06 
Baseline SOL scaled score 501.1 524.3 500.7 
Baseline SOL percent proficient or advanced 99 100 99 
Baseline SOL percent advanced 47 67 47 
Number of students 28,296 437 27,859 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B9. Descriptive statistics on student demographics and baseline scores, Advanced Placement history 
and social science courses 
Students who take the AP exam 


Characteristic oW-e-11 NAVAN Face to face 
Grade 9 3% 2% 3% 
Grade 10 26% 11% 26% 
Grade 11 38% 31% 38% 
Grade 12 33% 56% 33% 
Native American 0% 0% 0% 
Asian 15% 12% 15% 
Black or African American 10% 6% 10% 
Hispanic 8% 4% 8% 
White 62% 73% 61% 
Pacific Islander 0% 0% 0% 
Multiple races 5% 4% 5% 
Male 44% 42% 44% 
Age 16.6 16.9 16.5 
Economically disadvantaged 13% 14% 13% 
Gifted 40% 57% 40% 
English learner 2% 1% 2% 
Receives special education services 4% 2% 4% 
Region percent rural 53 59 53 
Baseline SOL z-score 0.09 0.42 0.08 
Baseline SOL scaled score 490.0 511.9 489.5 
Baseline SOL percent proficient or advanced 98 100 97 
Baseline SOL percent advanced 39 55 38 
Number of students 109,117 2,143 106,974 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B10. Descriptive statistics on student demographics and baseline scores, Advanced Placement 
languages courses 
Students who take the AP exam 


Characteristic oie} WAVE Face to face 
Grade 9 1% 0% 1% 
Grade 10 12% 17% 12% 
Grade 11 37% 36% 37% 
Grade 12 51% 47% 51% 
Native American 0% 1% 0% 
Asian 17% 35% 16% 
Black or African American 4% 4% 4% 
Hispanic 27% 4% 28% 
White 47% 49% 47% 
Pacific Islander 0% 1% 0% 
Multiple races 4% 6% 4% 
Male 37% 40% 36% 
Age 16.9 16.8 17.0 
Economically disadvantaged 20% 21% 20% 
Gifted 45% 57% 44% 
English learner 9% 7% 9% 
Receives special education services 3% 3% 3% 
Region percent rural 48 47 49 
Baseline SOL z-score 0.18 0.60 0.16 
Baseline SOL scaled score 483.8 518.3 482.5 
Baseline SOL percent proficient or advanced 94 98 94 
Baseline SOL percent advanced 39 61 38 
Number of students 3,553 136 3,417 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Full model results for Standards of Learning outcomes 

The coefficients (estimates) and standard errors for all variables included in the models used to estimate the 
relationship between taking a course online through Virtual Virginia (VVA) and SOL exam outcomes are in exhibits 
B11—B13. The first row of each exhibit displays the coefficient of interest: the relationship between VVA and the 
test outcome. Subsequent rows display each of the control variables in the model and the number of enrollments 
in that model. These models counted number of enrollments rather than number of students because a student 
may have been enrolled in more than one course in a subject area. (Appendix C presents the results of a sensitivity 
test aggregating results to the student level.) 


Standards of Learning test scores 
This section includes the linear regression results estimating the relationship between VVA and SOL z-scores. 
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Exhibit B11. Full model results for the relationship between Virtual Virginia and Standards of Learning score 
outcomes — Mathematics 


Estimate SE 

Virtual Virginia course -0.40 0.04 
Baseline score 0.47 <0.01 
Grade 9 -0.31 0.03 
Grade 10 -0.37 0.04 
Grade 11 -0.43 0.06 
Grade 12 -0.44 0.08 
Outcome test is algebra Il <0.01 0.02 
Outcome test is geometry 0.14 0.02 
Outcome year is 2016/17 0.03 0.02 
Student age -0.10 <0.01 
Male -0.02 <0.01 
Black/African American -0.14 0.01 
Hispanic -0.05 0.01 
Asian 0.19 0.01 
Economically disadvantaged -0.12 0.01 
Gifted 0.27 0.01 
English learner <0.01 0.01 
Receives special education services -0.13 0.01 
Pretest is algebra | 0.50 0.03 
Pretest is geometry 0.80 0.04 
Pretest is mathematics 8 0.09 0.03 
Pretest year is 2012 0.08 0.02 
Pretest year is 2013 0.20 0.04 
Pretest year is 2014 0.32 0.05 
Pretest year is 2015 0.34 0.07 
District percent rural -0.09 0.02 
Imputed for race, age, economic disadvantage, gifted, special education status, and male 0.09 0.20 
Imputed for English learner status -0.98 0.21 
Imputed for baseline score 0.10 0.01 
Constant -0.26 0.02 
Number of enrollments 

Virtual Virginia 1,765 

Face to face 381,970 

Total 383,735 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1.A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


B-11 


Exhibit B12. Full model results for the relationship between Virtual Virginia and Standards of Learning score 


outcomes — English language arts 


Estimate SE 

Virtual Virginia course -0.05 0.03 
Baseline score 0.58 0.01 
Grade 10 0.23 0.07 
Grade 11 0.45 0.07 
Grade 12 0.39 0.07 
Outcome year is 2016/17 0.02 0.02 
Student age -0.16 0.01 
Male -0.01 <0.01 
Black/African American -0.19 0.01 
Hispanic -0.03 0.01 
Asian 0.01 0.02 
Economically disadvantaged -0.14 0.01 
Gifted 0.27 0.01 
English learner -0.24 0.02 
Receives special education services -0.23 0.01 
Pretest year is 2012 0.08 0.03 
Pretest year is 2013 0.08 0.04 
Pretest year is 2014 0.10 0.06 
Pretest year is 2015 0.55 0.14 
District percent rural -0.06 0.02 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.01 0.25 
Imputed for English learner status -0.77 0.28 
Imputed for baseline score -0.07 0.01 
Constant 0.05 0.04 
Number of enrollments 

Virtual Virginia 964 

Face to face 128,049 

Total 129,013 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1.A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 


student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit B13. Full model results for the relationship between Virtual Virginia and Standards of Learning score 
outcomes — Science 


Estimate SE 

Virtual Virginia course -0.32 0.03 
Baseline score 0.61 0.01 
Grade 10 0.18 0.01 
Grade 11 0.20 0.03 
Grade 12 0.30 0.04 
Outcome test is chemistry -0.21 0.01 
Outcome test is earth science 0.19 0.01 
Outcome year is 2016/17 -0.03 0.01 
Student age -0.13 <0.01 
Male -0.01 <0.01 
Black/African American -0.18 0.01 
Hispanic -0.07 0.01 
Asian 0.14 0.01 
Economically disadvantaged -0.13 0.01 
Gifted 0.30 0.01 
English learner -0.11 0.01 
Receives special education services -0.19 0.01 
Pretest year is 2012 0.19 0.02 
Pretest year is 2013 0.30 0.03 
Pretest year is 2014 0.38 0.04 
Pretest year is 2015 0.44 0.05 
District percent rural -0.07 0.01 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.17 0.37 
Imputed for English learner status -0.56 0.38 
Imputed for baseline score <0.01 0.01 
Constant -0.05 0.01 
Number of enrollments 

Virtual Virginia 1,874 

Face to face 417,792 

Total 419,666 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1.A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


B-13 


Standards of Learning proficiency 


The logistic regression results estimating the relationship between VVA and proficiency on the SOL exam (a score 
of 400 or higher) are in exhibits B14—B16. 


Exhibit B14. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes — Mathematics 


Estimate SE 

Virtual Virginia course -1.42 0.14 
Baseline score 1.52 0.02 
Grade 9 -0.74 0.10 
Grade 10 -0.68 0.12 
Grade 11 -0.49 0.15 
Grade 12 -0.17 0.19 
Outcome test is algebra Il 0.90 0.06 
Outcome test is geometry 0.23 0.05 
Outcome year is 2016/17 -0.28 0.05 
Student age -0.32 0.01 
Male -0.09 0.01 
Black/African American -0.51 0.04 
Hispanic -0.18 0.04 
Asian 0.45 0.04 
Economically disadvantaged -0.42 0.02 
Gifted 0.98 0.04 
English learner -0.09 0.03 
Receives special education services -0.59 0.03 
Pretest is algebra | 0.86 0.11 
Pretest is geometry 0.93 0.15 
Pretest is mathematics 8 0.44 0.10 
Pretest year is 2012 0.27 0.06 
Pretest year is 2013 0.62 0.10 
Pretest year is 2014 0.88 0.13 
Pretest year is 2015 1.20 0.18 
District percent rural -0.25 0.06 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.37 0.57 
Imputed for English learner status -1.45 0.64 
Imputed for baseline score -0.04 0.02 
Constant -0.54 0.07 
Number of enrollments 

Virtual Virginia 1,765 

Face to face 381,970 

Total 383,735 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 


0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit B15. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes — English language arts 


Estimate SE 

Virtual Virginia course -0.21 0.12 
Baseline score 2.21 0.02 
Grade 10 0.77 0.18 
Grade 11 1.62 0.19 
Grade 12 1.64 0.22 
Outcome year is 2016/17 -0.10 0.06 
Student age -0.52 0.02 
Male -0.02 0.02 
Black/African American -0.62 0.04 
Hispanic 0.04 0.04 
Asian 0.22 0.06 
Economically disadvantaged -0.52 0.02 
Gifted 1.38 0.14 
English learner -0.87 0.05 
Receives special education services -1.04 0.03 
Pretest year is 2012 -0.20 0.09 
Pretest year is 2013 -0.18 0.13 
Pretest year is 2014 -0.27 0.19 
Pretest year is 2015 0.98 0.53 
District percent rural -0.19 0.05 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.10 0.76 
Imputed for English learner status -1.15 0.82 
Imputed for baseline score -0.33 0.04 
Constant 0.55 0.15 
Number of enrollments 

Virtual Virginia 964 

Face to face 128,049 

Total 129,013 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


B-15 


Exhibit B16. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes — Science 


Estimate SE 

Virtual Virginia course -1.08 0.09 
Baseline score 2.21 0.02 
Grade 10 0.87 0.05 
Grade 11 1.34 0.09 
Grade 12 1.94 0.13 
Outcome test is chemistry -0.06 0.04 
Outcome test is earth science 0.18 0.04 
Outcome year is 2016/17 -0.29 0.04 
Student age -0.49 0.01 
Male 0.04 0.01 
Black/African American -0.69 0.03 
Hispanic -0.25 0.03 
Asian 0.45 0.04 
Economically disadvantaged -0.53 0.02 
Gifted 1.58 0.08 
English learner -0.57 0.04 
Receives special education services -0.90 0.02 
Pretest year is 2012 0.21 0.08 
Pretest year is 2013 0.63 0.10 
Pretest year is 2014 1.02 0.13 
Pretest year is 2015 1.29 0.16 
District percent rural -0.05 0.04 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.66 0.53 
Imputed for English learner status -0.62 0.55 
Imputed for baseline score -0.04 0.04 
Constant 0.12 0.03 
Number of enrollments 

Virtual Virginia 1,874 

Face to face 417,792 

Total 419,666 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Full model results for Advanced Placement outcomes 

This section shows the coefficients (estimates) and standard errors for all variables included in the models used 
to estimate the relationship between VVA and AP exam outcomes. The first row displays the coefficient of interest: 
the relationship between VVA and the test outcome. Subsequent rows display each of the control variables in the 
model and the number of enrollments in that model. These models count number of enrollments rather than 
number of students because a student may have been enrolled in more than one course in a subject area. 
(Appendix C presents the results of a sensitivity test aggregating results to the student level.) 


Advanced Placement test scores 
The linear regression results estimating the relationship between VVA and AP z-scores are in exhibits B17—B21. 


B-16 


Exhibit B17. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes — Mathematics 


Estimate SE 

Virtual Virginia course -0.34 0.07 
Baseline score 0.39 0.01 
Grade 10 -0.06 0.11 
Grade 11 -0.12 0.14 
Grade 12 -0.01 0.16 
Outcome test is calculus AB -0.12 0.05 
Outcome test is calculus BC -0.31 0.05 
Outcome test is computer science -0.34 0.07 
Outcome year is 2016/17 -0.03 0.02 
Student age -0.14 0.02 
Male 0.13 0.01 
Black/African American -0.31 0.03 
Hispanic -0.15 0.03 
Asian 0.02 0.03 
Economically disadvantaged -0.21 0.03 
Gifted 0.19 0.02 
English learner 0.09 0.05 
Receives special education services 0.12 0.04 
Pretest is algebra | 0.42 0.04 
Pretest is geometry 0.82 0.06 
Pretest year is 2013 0.09 0.04 
Pretest year is 2014 0.31 0.07 
Pretest year is 2015 0.35 0.20 
District percent rural -0.11 0.06 
Imputed for baseline score <0.01 0.06 
Constant -0.43 0.06 
Number of enrollments 

Virtual Virginia 296 

Face to face 29,846 

Total 30,142 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit B18. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes — English language arts 


Estimate SE 

Virtual Virginia course -0.06 0.07 
Baseline score 0.40 0.01 
Grade 10 0.59 0.58 
Grade 11 0.80 0.62 
Grade 12 1.23 0.64 
Outcome subject is English language 0.31 0.13 
Outcome year is 2016/17 -0.04 0.02 
Student age -0.11 0.01 
Male -0.04 0.01 
Black/African American -0.34 0.03 
Hispanic -0.11 0.02 
Asian 0.14 0.03 
Economically disadvantaged -0.29 0.02 
Gifted 0.39 0.02 
English learner -0.10 0.05 
Receives special education services -0.03 0.03 
Pretest year is 2013 0.06 0.03 
Pretest year is 2014 0.06 0.23 
Pretest year is 2015 0.26 0.53 
District percent rural -0.19 0.05 
Imputed for baseline score 0.26 0.02 
Constant -0.17 0.06 
Number of enrollments 

Virtual Virginia 291 

Face to face 41,835 

Total 42,126 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B19. Full model results for the relationship between Virtual Virginia and Advanced Placement score 


outcomes — Science 


Estimate 

Virtual Virginia course -0.24 0.07 
Baseline score 0.47 0.01 
Grade 10 0.46 0.12 
Grade 11 0.49 0.12 
Grade 12 0.62 0.13 
Outcome test is biology 0.19 0.05 
Outcome test is environmental science 0.29 0.06 
Outcome year is 2016/17 0.04 0.03 
Student age -0.12 0.01 
Male 0.18 0.02 
Black/African American -0.41 0.03 
Hispanic -0.18 0.03 
Asian -0.02 0.03 
Economically disadvantaged -0.21 0.02 
Gifted 0.28 0.02 
English learner 0.15 0.05 
Receives special education services 0.01 0.03 
Pretest year is 2013 0.02 0.03 
Pretest year is 2014 0.08 0.06 
Pretest year is 2015 0.13 0.10 
District percent rural -0.21 0.06 
Imputed for baseline score 0.05 0.03 
Constant -0.16 0.04 
Number of enrollments 

Virtual Virginia 437 

Face to face 27,859 

Total 28,296 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B20. Full model output for the relationship between Virtual Virginia and Advanced Placement score 
outcomes — History and social science 


Estimate SE 

Virtual Virginia course -0.31 0.05 
Baseline score 0.40 0.01 
Grade 10 0.32 0.09 
Grade 11 0.48 0.10 
Grade 12 0.55 0.11 
Outcome test is art history -0.07 0.10 
Outcome test is European history -0.02 0.06 
Outcome test is comparative government 0.01 0.07 
Outcome test is U.S. government -0.06 0.06 
Outcome test is human geography 0.14 0.05 
Outcome test is macroeconomics -0.34 0.10 
Outcome test is microeconomics -0.22 0.08 
Outcome test is psychology 0.07 0.06 
Outcome test is U.S. history -0.07 0.06 
Outcome year is 2016/17 0.01 0.02 
Student age -0.10 0.01 
Male 0.13 0.01 
Black/African American -0.31 0.03 
Hispanic -0.13 0.02 
Asian 0.13 0.03 
Economically disadvantaged -0.27 0.02 
Gifted 0.34 0.01 
English learner -0.03 0.03 
Receives special education services -0.01 0.02 
Pretest year is 2013 -0.02 0.02 
Pretest year is 2014 -0.01 0.04 
Pretest year is 2015 -0.01 0.08 
District percent rural -0.14 0.05 
Imputed for baseline score 0.19 0.01 
Constant 0.04 0.04 
Number of enrollments 

Virtual Virginia 2,143 

Face to face 106,974 

Total 109,117 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit B21. Full model output for the relationship between Virtual Virginia and Advanced Placement score 
outcomes — Languages 


Estimate SE 

Virtual Virginia course -0.19 0.18 
Baseline score 0.28 0.02 
Grade 11 -0.09 0.20 
Grade 12 -0.04 0.22 
Outcome test is Chinese -0.19 0.16 
Outcome test is French 0.08 0.09 
Outcome test is Latin -0.02 0.13 
Outcome year is 2016/17 -0.12 0.19 
Student age -0.16 0.04 
Male -0.02 0.04 
Black/African American -0.25 0.08 
Hispanic 0.61 0.06 
Asian 0.11 0.11 
Economically disadvantaged -0.03 0.06 
Gifted 0.14 0.04 
English learner 0.36 0.07 
Receives special education services -0.27 0.08 
Pretest year is 2013 -0.26 0.14 
Pretest year is 2014 -0.16 0.22 
Pretest year is 2015 -0.08 0.24 
District percent rural -0.19 0.11 
Imputed for baseline score 0.33 0.05 
Constant -0.05 0.11 
Number of enrollments 

Virtual Virginia 136 

Face to face 3,417 

Total 3,553 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Advanced Placement proficiency 


The logistic regression results estimating the relationship between VVA and AP proficiency (a score of 3 or higher) 
are in exhibits B22—B26. 


Exhibit B22. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes — Mathematics 


Estimate SE 

Virtual Virginia course -0.81 0.18 
Baseline score 0.87 0.03 
Grade 10 -0.68 1.03 
Grade 11 -0.76 1.13 
Grade 12 -0.49 1.18 
Outcome test is calculus AB 0.18 0.11 
Outcome test is calculus BC 0.70 0.12 
Outcome test is computer science 0.04 0.16 
Outcome year is 2016/17 -0.21 0.05 
Student age -0.29 0.05 
Male 0.28 0.03 
Black/African American -0.71 0.09 
Hispanic -0.35 0.07 
Asian 0.04 0.07 
Economically disadvantaged -0.51 0.07 
Gifted 0.42 0.05 
English learner 0.20 0.14 
Receives special education services 0.36 0.11 
Pretest is algebra | 1.10 0.14 
Pretest is geometry 1.92 0.17 
Pretest year is 2013 0.25 0.09 
Pretest year is 2014 0.94 0.20 
Pretest year is 2015 0.26 1.58 
District percent rural -0.21 0.12 
Imputed for baseline score 0.05 0.15 
Constant -0.76 0.16 
Number of enrollments 

Virtual Virginia 296 

Face to face 29,846 

Total 30,142 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit B23. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes — English language arts 


Estimate SE 

Virtual Virginia course -0.02 0.16 
Baseline score 0.99 0.02 
Grade 10 -0.41 1.29 
Grade 11 0.22 1.36 
Grade 12 1.03 1.41 
Outcome subject is English language 0.58 0.26 
Outcome year is 2016/17 -0.16 0.06 
Student age -0.27 0.04 
Male -0.07 0.03 
Black/African American -0.74 0.07 
Hispanic -0.19 0.06 
Asian 0.37 0.08 
Economically disadvantaged -0.71 0.06 
Gifted 0.85 0.04 
English learner -0.54 0.13 
Receives special education services -0.15 0.08 
Pretest year is 2013 0.26 0.07 
Pretest year is 2014 -0.27 0.54 
Pretest year is 2015 -0.78 1.02 
District percent rural -0.36 0.11 
Imputed for baseline score 0.55 0.05 
Constant 0.37 0.13 
Number of enrollments 

Virtual Virginia 291 

Face to face 41,835 

Total 42,126 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B24. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes — Science 


Estimate SE 

Virtual Virginia course -0.67 0.17 
Baseline score 1.13 0.03 
Grade 10 0.98 0.25 
Grade 11 1.20 0.34 
Grade 12 1.50 0.37 
Outcome test is biology 2.19 0.13 
Outcome test is environmental science 1.34 0.15 
Outcome year is 2016/17 0.37 0.07 
Student age -0.28 0.04 
Male 0.45 0.04 
Black/African American -1.13 0.08 
Hispanic -0.44 0.08 
Asian -0.08 0.05 
Economically disadvantaged -0.59 0.07 
Gifted 0.64 0.05 
English learner 0.25 0.16 
Receives special education services <0.01 0.08 
Pretest year is 2013 -0.05 0.08 
Pretest year is 2014 0.12 0.18 
Pretest year is 2015 0.20 0.28 
District percent rural -0.41 0.12 
Imputed for baseline score 0.17 0.08 
Constant -1.02 0.11 
Number of enrollments 

Virtual Virginia 456 

Face to face 28,022 

Total 28,478 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit B25. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes — History and social science 


Estimate SE 

Virtual Virginia course -0.76 0.10 
Baseline score 0.94 0.02 
Grade 10 0.68 0.22 
Grade 11 1.08 0.25 
Grade 12 1.07 0.26 
Outcome test is art history -0.04 0.26 
Outcome test is European history -0.90 0.14 
Outcome test is comparative government -0.66 0.17 
Outcome test is U.S. government -0.37 0.13 
Outcome test is human geography -0.09 0.12 
Outcome test is macroeconomics -0.48 0.22 
Outcome test is microeconomics -0.18 0.20 
Outcome test is psychology 0.25 0.13 
Outcome test is U.S. history -0.61 0.14 
Outcome year is 2016/17 0.11 0.04 
Student age -0.22 0.02 
Male 0.33 0.03 
Black/African American -0.69 0.06 
Hispanic -0.31 0.05 
Asian 0.26 0.06 
Economically disadvantaged -0.56 0.05 
Gifted 0.70 0.03 
English learner -0.04 0.08 
Receives special education services -0.03 0.05 
Pretest year is 2013 -0.20 0.05 
Pretest year is 2014 -0.11 0.10 
Pretest year is 2015 -0.25 0.20 
District percent rural -0.24 0.11 
Imputed for baseline score 0.38 0.04 
Constant 0.93 0.09 
Number of enrollments 

Virtual Virginia 2,143 

Face to face 106,974 

Total 109,117 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit B26. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes — Languages 


Estimate SE 

Virtual Virginia course -0.47 0.40 
Baseline score 0.65 0.08 
Grade 11 0.06 0.99 
Grade 12 0.54 0.54 
Outcome test is Chinese -0.35 0.59 
Outcome test is French -1.26 0.24 
Outcome test is Latin -2.11 0.28 
Outcome year is 2016/17 -0.89 0.65 
Student age -0.37 0.14 
Male -0.21 0.11 
Black/African American -0.87 0.22 
Hispanic 1.17 0.23 
Asian 0.41 0.22 
Economically disadvantaged -0.18 0.18 
Gifted 0.23 0.15 
English learner 1.35 0.39 
Receives special education services -0.46 0.25 
Pretest year is 2013 0.39 0.78 
Pretest year is 2014 -12.20 1.32 
District percent rural -0.34 0.25 
Imputed for baseline score 0.58 0.23 
Constant 3.35 0.26 
Number of enrollments 

Virtual Virginia 136 

Face to face 3,417 

Total 3,553 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Appendix C: Sensitivity analyses 

To test the sensitivity of the estimates shown in appendix B to certain model specifications, the study team ran 
the same models using four alternative specifications: using a student’s mean score within subject area as the 
outcome, adding division fixed effects, using alternative versions of the demographic variables used as controls, 
and adding interactions between student grade level and the outcome course. 


A summary of the results from these sensitivity tests is shown in exhibit C1. Detailed results and explanations of 
the tests are shown in the subsequent sections. 


Exhibit C1. Summary of sensitivity tests on the relationship between Virtual Virginia and test outcomes 
SOL exam AP exam 
scores SOL proficiency scores AP proficiency 


Mathematics 


Main specification -0.40 -1.42 -0.34 -0.81 
Mean score outcome -0.44 -1.61 -0.31 -0.48 
Division fixed effects -0.39 -1.42 -0.26 -0.68 
Alternate versions of demographic variables -0.39 -1.40 -0.33 -0.76 
Interaction between grade level and baseline course -0.46 -1.47 -0.34 -0.80 
English language arts 

Main specification -0.05 -0.21 -0.06 -0.02 
Mean score outcome -0.05 -0.21 -0.06 -0.03 
Division fixed effects -0.05 -0.27 0.02 0.19 
Alternate versions of demographic variables -0.04 -0.24 -0.01 0.10 
Interaction between grade level and baseline course N/A N/A -0.06 -0.01 
Science 

Main specification -0.32 -1.08 -0.24 -0.67 
Mean score outcome -0.25 -1.04 -0.27 -0.82 
Division fixed effects -0.30 -1.05 -0.22 -0.68 
Alternate versions of demographic variables -0.31 -1.06 -0.22 -0.62 
Interaction between grade level and baseline course -0.34 -1.17 -0.27 -0.70 
History and social science 

Main specification N/A N/A -0.31 -0.76 
Mean score outcome N/A N/A -0.31 -0.75 
Division fixed effects N/A N/A -0.08 -0.24 
Alternate versions of demographic variables N/A N/A -0.28 -0.67 
Interaction between grade level and baseline course N/A N/A -0.34 -0.82 
Languages 

Main specification N/A N/A -0.19 -0.47 
Mean score outcome N/A N/A -0.29 -0.58 
Division fixed effects N/A N/A -0.02 -0.33 
Alternate versions of demographic variables N/A N/A -0.24 -0.51 
Interaction between grade level and baseline course N/A N/A -0.19 -0.49 


N/A is not applicable. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Mean score within subject area 

In the main analysis, test scores were each treated independently, even when students took more than one test 
in the same subject area in the same year. That is, students who took both the AP U.S. history and the AP world 
history exam in the same year had two records contributing to the estimates of the relationship between VVA 


and AP exam scores in history and social science. This sensitivity analysis examined an alternative approach in 
which we used mean test scores within each subject area. In the example above, the student’s two AP exam 
scores were averaged, and she appeared only once in this analysis. No substantive differences were found in the 
estimates of the relationship between VVA and test score outcomes. 


Standards of Learning outcomes 


Results of the sensitivity analyses for courses with SOL exams follow. 


Standards of Learning exam scores 


The results for the sensitivity analyses based on the z-score translation of continuous SOL exam scores are in 
exhibits C2 to C4. 


Exhibit C2. Full model results for the relationship between Virtual Virginia and mean Standards of Learning 
score outcomes within area —- Mathematics 


Estimate SE 

Virtual Virginia course -0.44 0.04 
Mean baseline score 0.45 0.01 
Grade 9 0.06 0.02 
Grade 10 0.13 0.04 
Grade 11 0.09 0.05 
Grade 12 0.21 0.07 
Outcome year is 2016/17 -0.12 0.02 
Student age -0.10 <0.01 
Male -0.02 <0.01 
Black/African American -0.16 0.01 
Hispanic -0.06 0.01 
Asian 0.24 0.01 
Economically disadvantaged -0.17 0.01 
Gifted 0.43 0.01 
English learner -0.03 0.01 
Receives special education services -0.19 0.01 
Pretest year is 2012 0.20 0.02 
Pretest year is 2013 0.44 0.04 
Pretest year is 2014 0.70 0.05 
Pretest year is 2015 0.88 0.07 
District percent rural -0.09 0.02 
Imputed for race, age, economic disadvantage, gifted, special education status, and male 0.10 0.20 
Imputed for English learner status -1.00 0.21 
Imputed for baseline score -0.04 0.01 
Constant -0.13 0.01 
Number of students 

Virtual Virginia 1,681 

Face to face 377,050 

Total 378,731 


Note: This analysis uses z-scores as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores are created by taking each student's 
SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A negative 
number indicates that the student is below the average for his or her grade level and school year, while a positive number indicates that a student is above 
average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C3. Full model results for the relationship between Virtual Virginia and mean Standards of Learning 


score outcomes within area — English language arts 


Estimate SE 

Virtual Virginia course -0.05 0.03 
Baseline score 0.58 <0.01 
Grade 10 0.22 0.07 
Grade 11 0.45 0.07 
Grade 12 0.39 0.07 
Outcome year is 2016/17 0.02 0.02 
Student age -0.16 0.01 
Male -0.01 <0.01 
Black/African American -0.19 0.01 
Hispanic -0.03 0.01 
Asian 0.01 0.01 
Economically disadvantaged -0.14 0.01 
Gifted 0.27 0.01 
English learner -0.24 0.02 
Receives special education services -0.23 0.01 
Pretest year is 2012 0.08 0.03 
Pretest year is 2013 0.08 0.04 
Pretest year is 2014 0.10 0.06 
Pretest year is 2015 0.55 0.14 
District percent rural -0.06 0.02 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.01 0.25 
Imputed for English learner status -0.77 0.27 
Imputed for baseline score -0.07 0.01 
Constant 0.04 0.04 
Number of students 

Virtual Virginia 964 

Face to face 128,049 

Total 129,013 


Note: This analysis uses z-scores as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores are created by taking each student's 
SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A negative 
number indicates that the student is below the average for his or her grade level and school year, while a positive number indicates that a student is above 


average for his or her grade level and school year. Average z-scores for the overall population are always zero. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C4. Full model results for the relationship between Virtual Virginia and mean Standards of Learning 


score outcomes within area — Science 


Estimate SE 

Virtual Virginia course -0.25 0.03 
Baseline score 0.59 0.01 
Grade 10 0.06 0.01 
Grade 11 <0.01 0.03 
Grade 12 0.15 0.04 
Outcome year is 2016/17 -0.08 0.01 
Student age -0.11 <0.01 
Male <0.01 <0.01 
Black/African American -0.18 0.01 
Hispanic -0.08 0.01 
Asian 0.09 0.01 
Economically disadvantaged -0.11 0.01 
Gifted 0.27 0.01 
English learner -0.11 0.01 
Receives special education services -0.18 0.01 
Pretest year is 2012 0.21 0.02 
Pretest year is 2013 0.36 0.03 
Pretest year is 2014 0.50 0.04 
Pretest year is 2015 0.60 0.05 
District percent rural -0.04 0.01 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.20 0.42 
Imputed for English learner status -0.45 0.42 
Imputed for baseline score -0.03 0.01 
Constant -0.06 0.01 
Number of students 

Virtual Virginia 1,757 

Face to face 410,833 

Total 412,590 


Note: This analysis uses z-scores as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores are created by taking each student's 
SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A negative 
number indicates that the student is below the average for his or her grade level and school year, while a positive number indicates that a student is above 


average for his or her grade level and school year. Average z-scores for the overall population are always zero. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Standards of Learning proficiency 


The results for the sensitivity analyses based on the dichotomization of SOL exam scores are in exhibits C5 to C8. 


Exhibit C5. Full model results for the relationship between Virtual Virginia and mean Standards of Learning 


proficiency outcomes — Mathematics 


Estimate SE 

Virtual Virginia course -1.61 0.13 
Baseline score 1.66 0.02 
Grade 9 -0.10 0.07 
Grade 10 0.47 0.08 
Grade 11 1.11 0.12 
Grade 12 1.70 0.16 
Outcome year is 2016/17 -0.48 0.05 
Student age -0.35 0.01 
Male -0.10 0.01 
Black/African American -0.54 0.04 
Hispanic -0.19 0.04 
Asian 0.53 0.04 
Economically disadvantaged -0.51 0.02 
Gifted 1.28 0.04 
English learner -0.18 0.03 
Receives special education services -0.70 0.03 
Pretest year is 2012 0.46 0.06 
Pretest year is 2013 0.99 0.10 
Pretest year is 2014 1.45 0.13 
Pretest year is 2015 1.98 0.17 
District percent rural -0.25 0.06 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.22 0.62 
Imputed for English learner status -1.68 0.68 
Imputed for baseline score -0.22 0.02 
Constant -0.11 0.03 
Number of enrollments 

Virtual Virginia 1,681 

Face to face 377,050 

Total 378,731 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 


0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C6. Full model results for the relationship between Virtual Virginia and mean Standards of Learning 
proficiency outcomes — English language arts 


Estimate SE 

Virtual Virginia course -0.21 0.12 
Baseline score 2.21 0.02 
Grade 10 0.77 0.18 
Grade 11 1.62 0.19 
Grade 12 1.64 0.22 
Outcome year is 2016/17 -0.10 0.06 
Student age -0.52 0.02 
Male -0.02 0.02 
Black/African American -0.62 0.04 
Hispanic 0.04 0.04 
Asian 0.22 0.06 
Economically disadvantaged -0.52 0.02 
Gifted 1.38 0.14 
English learner -0.87 0.05 
Receives special education services -1.04 0.03 
Pretest year is 2012 -0.20 0.09 
Pretest year is 2013 -0.18 0.13 
Pretest year is 2014 -0.27 0.19 
Pretest year is 2015 0.98 0.53 
District percent rural -0.19 0.05 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.10 0.76 
Imputed for English learner status -1.15 0.82 
Imputed for baseline score -0.33 0.04 
Constant 0.55 0.15 
Number of enrollments 

Virtual Virginia 964 

Face to face 128,049 

Total 129,013 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C7. Full model results for the relationship between Virtual Virginia and mean Standards of Learning 
proficiency outcomes — Science 


Estimate SE 

Virtual Virginia course -1.04 0.09 
Baseline score 2.19 0.02 
Grade 10 0.80 0.04 
Grade 11 1.26 0.09 
Grade 12 1.89 0.13 
Outcome year is 2016/17 -0.32 0.04 
Student age -0.48 0.01 
Male 0.04 0.01 
Black/African American -0.68 0.03 
Hispanic -0.26 0.03 
Asian 0.42 0.04 
Economically disadvantaged -0.51 0.02 
Gifted 1.56 0.08 
English learner -0.57 0.04 
Receives special education services -0.90 0.02 
Pretest year is 2012 0.24 0.08 
Pretest year is 2013 0.69 0.11 
Pretest year is 2014 1.13 0.13 
Pretest year is 2015 1.43 0.16 
District percent rural -0.04 0.04 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.50 0.58 
Imputed for English learner status -0.80 0.58 
Imputed for baseline score -0.05 0.04 
Constant 0.14 0.03 
Number of enrollments 

Virtual Virginia 1,757 

Face to face 410,833 

Total 412,590 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Advanced Placement outcomes 
Results of the sensitivity analyses using the mean AP exam score at the student level follow. 


Advanced Placement test scores 


The results for the sensitivity analyses based on the z-score translation of AP exam scores at the student level are 
in exhibits C8 to C12. 


Exhibit C8. Full model results for the relationship between Virtual Virginia and mean Advanced Placement 
score outcomes — Mathematics 


Estimate SE 

Virtual Virginia course -0.31 0.07 
Baseline score 0.36 0.01 
Grade 10 -0.05 0.10 
Grade 11 -0.09 0.12 
Grade 12 -0.14 0.14 
Outcome year is 2016/17 0.01 0.02 
Student age -0.14 0.02 
Male 0.09 0.01 
Black/African American -0.32 0.03 
Hispanic -0.19 0.03 
Asian 0.03 0.03 
Economically disadvantaged -0.23 0.03 
Gifted 0.31 0.02 
English learner -0.05 0.05 
Receives special education services 0.05 0.03 
Pretest year is 2013 0.04 0.04 
Pretest year is 2014 0.15 0.07 
Pretest year is 2015 0.05 0.17 
District percent rural -0.12 0.06 
Imputed for baseline score 0.17 0.05 
Constant -0.01 0.04 
Number of students 

Virtual Virginia 273 

Face to face 28,684 

Total 28,957 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C9. Full model results for the relationship between Virtual Virginia and mean Advanced Placement 
score outcomes — English language arts 


Estimate SE 

Virtual Virginia course -0.06 0.06 
Baseline score 0.40 0.01 
Grade 10 <0.01 <0.01 
Grade 11 0.20 0.23 
Grade 12 0.36 0.24 
Outcome year is 2016/17 -0.05 0.03 
Student age -0.12 0.02 
Male -0.05 0.02 
Black/African American -0.32 0.03 
Hispanic -0.11 0.02 
Asian 0.16 0.04 
Economically disadvantaged -0.29 0.02 
Gifted 0.41 0.02 
English learner -0.02 0.04 
Receives special education services -0.04 0.03 
Pretest year is 2013 0.07 0.03 
Pretest year is 2014 0.09 0.36 
Pretest year is 2015 <0.01 <0.01 
District percent rural -0.20 0.06 
Imputed for baseline score -0.10 0.10 
Constant -0.08 0.04 
Number of enrollments 

Virtual Virginia 257 

Face to face 37,122 

Total 37,379 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C10. Full model results for the relationship between Virtual Virginia and mean Advanced Placement 
score outcomes — Science 


Estimate SE 

Virtual Virginia course -0.27 0.06 
Baseline score 0.49 0.01 
Grade 10 0.44 0.15 
Grade 11 0.46 0.13 
Grade 12 0.69 0.14 
Outcome year is 2016/17 0.03 0.03 
Student age -0.11 0.02 
Male 0.11 0.02 
Black/African American -0.39 0.03 
Hispanic -0.21 0.03 
Asian -0.06 0.04 
Economically disadvantaged -0.19 0.02 
Gifted 0.26 0.02 
English learner 0.10 0.05 
Receives special education services 0.05 0.03 
Pretest year is 2013 0.10 0.04 
Pretest year is 2014 0.19 0.08 
Pretest year is 2015 0.22 0.11 
District percent rural -0.17 0.06 
Imputed for baseline score 0.01 0.13 
Constant -0.02 0.04 
Number of students 

Virtual Virginia 328 

Face to face 20,534 

Total 20,862 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C11. Full model output for the relationship between Virtual Virginia and mean Advanced Placement 
score outcomes — History and social science 


Estimate SE 

Virtual Virginia course -0.31 0.04 
Baseline score 0.39 0.01 
Grade 10 0.21 0.08 
Grade 11 0.31 0.10 
Grade 12 0.39 0.11 
Outcome year is 2016/17 0.02 0.02 
Student age -0.09 0.01 
Male 0.11 0.01 
Black/African American -0.31 0.03 
Hispanic -0.12 0.02 
Asian 0.14 0.02 
Economically disadvantaged -0.27 0.02 
Gifted 0.34 0.01 
English learner 0.02 0.03 
Receives special education services 0.01 0.02 
Pretest year is 2013 -0.02 0.02 
Pretest year is 2014 -0.02 0.05 
Pretest year is 2015 -0.04 0.08 
District percent rural -0.12 0.05 
Imputed for baseline score 0.03 0.08 
Constant -0.03 0.03 
Number of students 

Virtual Virginia 1,360 

Face to face 83,327 

Total 84,687 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C12. Full model output for the relationship between Virtual Virginia and mean Advanced Placement 
score outcomes — Languages 


Estimate SE 

Virtual Virginia course -0.29 0.20 
Baseline score 0.26 0.02 
Grade 10 -0.85 0.48 
Grade 11 -0.55 0.37 
Grade 12 -0.16 0.28 
Outcome year is 2016/17 -0.49 0.24 
Student age -0.16 0.05 
Male -0.05 0.04 
Black/African American -0.34 0.09 
Hispanic 0.62 0.07 
Asian 0.07 0.10 
Economically disadvantaged -0.03 0.06 
Gifted 0.16 0.04 
English learner 0.31 0.09 
Receives special education services -0.23 0.09 
Pretest year is 2013 0.44 0.26 
Pretest year is 2014 0.93 0.45 
District percent rural -0.20 0.11 
Imputed for baseline score 0.41 0.31 
Constant <0.01 0.12 
Number of students 

Virtual Virginia 118 

Face to face 2,982 

Total 3,100 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Advanced Placement proficiency 


The results for the sensitivity analyses based on the dichotomization of AP exam scores are in exhibits C13 to C17. 


Exhibit C13. Full model output for the relationship between Virtual Virginia and mean Advanced Placement 
proficiency outcomes — Mathematics 


Estimate SE 

Virtual Virginia course -0.48 0.17 
Baseline score 0.82 0.04 
Grade 10 -1.14 1.02 
Grade 11 -1.18 1.15 
Grade 12 -1.33 1.20 
Outcome year is 2016/17 -0.14 0.05 
Student age -0.30 0.05 
Male 0.25 0.03 
Black/African American -0.75 0.09 
Hispanic -0.39 0.07 
Asian 0.09 0.08 
Economically disadvantaged -0.58 0.07 
Gifted 0.69 0.04 
English learner -0.11 0.14 
Receives special education services 0.24 0.10 
Pretest year is 2013 0.15 0.09 
Pretest year is 2014 0.71 0.20 
Pretest year is 2015 -0.31 1.46 
District percent rural -0.08 0.13 
Imputed for baseline score 0.39 0.14 
Constant 0.59 0.09 
Number of enrollments 

Virtual Virginia 273 

Face to face 28,684 

Total 28,957 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C14. Full model output for the relationship between Virtual Virginia and mean Advanced Placement 
proficiency outcomes — English language arts 


Estimate SE 

Virtual Virginia course -0.03 0.17 
Baseline score 1.01 0.02 
Grade 10 -0.92 0.60 
Grade 11 -0.28 0.08 
Outcome year is 2016/17 -0.20 0.07 
Student age -0.30 0.04 
Male -0.09 0.04 
Black/African American -0.69 0.07 
Hispanic -0.21 0.06 
Asian 0.39 0.09 
Economically disadvantaged -0.71 0.06 
Gifted 0.87 0.04 
English learner -0.36 0.12 
Receives special education services -0.16 0.09 
Pretest year is 2013 0.28 0.08 
Pretest year is 2014 -0.48 0.81 
District percent rural -0.37 0.12 
Imputed for baseline score -0.22 0.29 
Constant 0.49 0.09 
Number of enrollments 

Virtual Virginia 257 

Face to face 37,122 

Total 37,379 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C15. Full model output for the relationship between Virtual Virginia and mean Advanced Placement 
proficiency outcomes — Science 


Estimate SE 

Virtual Virginia course -0.82 0.19 
Baseline score 1.12 0.03 
Grade 10 1.66 0.30 
Grade 11 1.22 0.41 
Grade 12 2.00 0.43 
Outcome year is 2016/17 0.24 0.09 
Student age -0.24 0.05 
Male 0.06 0.05 
Black/African American -0.95 0.10 
Hispanic -0.45 0.09 
Asian -0.08 0.07 
Economically disadvantaged -0.50 0.07 
Gifted 0.42 0.06 
English learner 0.06 0.17 
Receives special education services 0.04 0.09 
Pretest year is 2013 0.21 0.11 
Pretest year is 2014 0.48 0.21 
Pretest year is 2015 0.55 0.35 
District percent rural -0.03 0.15 
Imputed for baseline score 0.20 0.43 
Constant 0.12 0.09 
Number of enrollments 

Virtual Virginia 328 

Face to face 20,534 

Total 20,862 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C16. Full model output for the relationship between Virtual Virginia and mean Advanced Placement 
proficiency outcomes — History and social science 


Estimate SE 

Virtual Virginia course -0.75 0.10 
Baseline score 0.91 0.02 
Grade 10 0.61 0.20 
Grade 11 0.71 0.23 
Grade 12 0.80 0.25 
Outcome year is 2016/17 0.12 0.05 
Student age -0.21 0.02 
Male 0.27 0.03 
Black/African American -0.67 0.07 
Hispanic -0.27 0.06 
Asian 0.34 0.06 
Economically disadvantaged -0.57 0.05 
Gifted 0.68 0.03 
English learner 0.02 0.08 
Receives special education services 0.03 0.05 
Pretest year is 2013 -0.23 0.06 
Pretest year is 2014 -0.18 0.12 
Pretest year is 2015 -0.34 0.22 
District percent rural -0.18 0.12 
Imputed for baseline score -0.02 0.21 
Constant 0.61 0.07 
Number of enrollments 

Virtual Virginia 1,360 

Face to face 83,327 

Total 84,687 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C17. Full model output for the relationship between Virtual Virginia and mean Advanced Placement 
proficiency outcomes — Languages 


Estimate SE 

Virtual Virginia course -0.58 0.41 
Baseline score 0.51 0.08 
Grade 10 13.51 1.43 
Grade 11 0.85 1.10 
Grade 12 1.11 0.74 
Outcome year is 2016/17 -0.64 0.67 
Student age -0.28 0.15 
Male -0.23 0.13 
Black/African American -0.86 0.23 
Hispanic 1.63 0.22 
Asian 0.34 0.23 
Economically disadvantaged -0.06 0.18 
Gifted 0.16 0.14 
English learner 1.01 0.39 
Receives special education services -0.50 0.27 
Pretest year is 2013 0.15 0.76 
Pretest year is 2014 -11.61 1.28 
District percent rural -0.52 0.22 
Imputed for baseline score <0.01 <0.01 
Constant 2.50 0.25 
Number of enrollments 

Virtual Virginia 118 

Face to face 2,973 

Total 3,091 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Division fixed effects 

The main models reported in appendix B were single-level models with standard errors clustered at the school 
level. This alternative analysis added district-level fixed effects to account for all time-invariant preexisting 
differences between districts. This alternative specification did not substantially change the estimates on SOL 
proficiency or scaled score outcomes. On the AP exam outcomes, the additional control for district fixed effects 
reduced the size of the estimates so that the relationship between VVA and AP exam scores in history and social 
science was no longer substantively important. The relationship between VVA and other AP exam outcomes did 
not substantially change. Thus, it appears that some portion of the relationship between VVA and AP exam 
outcomes in history and social science may be explained by preexisting differences between districts. 
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Standards of Learning outcomes 
Results of the sensitivity analyses for courses with SOL exams follow. 


Standards of Learning test scores 


The results for the sensitivity analyses based on the z-score translation of continuous SOL exam scores are in 
exhibits C18 to C20. 


Exhibit C18. Full model results for the relationship between Virtual Virginia and Standards of Learning score 
outcomes with district fixed effects - Mathematics 


Estimate SE 

Virtual Virginia course -0.39 0.04 
Baseline score 0.46 <0.01 
Grade 9 -0.31 0.03 
Grade 10 -0.39 0.04 
Grade 11 -0.47 0.06 
Grade 12 -0.48 0.08 
Outcome test is algebra Il <0.01 0.02 
Outcome test is geometry 0.14 0.02 
Outcome year is 2016/17 0.04 0.02 
Student age -0.09 <0.01 
Male -0.02 <0.01 
Black/African American -0.12 0.01 
Hispanic -0.05 0.01 
Asian 0.17 0.01 
Economically disadvantaged -0.11 <0.01 
Gifted 0.26 0.01 
English learner -0.02 0.01 
Receives special education services -0.14 0.01 
Pretest is geometry 0.35 0.02 
Pretest is mathematics 6 -0.49 0.03 
Pretest is mathematics 7 -0.42 0.01 
Pretest year is 2012 0.07 0.02 
Pretest year is 2013 0.17 0.04 
Pretest year is 2014 0.28 0.05 
Pretest year is 2015 0.29 0.07 
Imputed for race, age, economic disadvantage, gifted, special education status, and male 0.07 0.10 
Imputed for English learner status -0.01 0.17 
Imputed for baseline score -0.70 0.39 
Constant -0.02 0.04 
Number of enrollments 

Virtual Virginia 1,765 

Face to face 381,970 

Total 383,735 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C19. Full model results for the relationship between Virtual Virginia and Standards of Learning score 


outcomes with district fixed effects — English language arts 


Estimate SE 

Virtual Virginia course -0.05 0.03 
Baseline score 0.57 <0.01 
Grade 10 0.27 0.05 
Grade 11 0.49 0.05 
Grade 12 0.46 0.05 
Outcome year is 2016/17 <0.01 0.02 
Student age -0.16 0.01 
Male -0.01 <0.01 
Black/African American -0.19 0.01 
Hispanic -0.07 0.01 
Asian -0.06 0.01 
Economically disadvantaged -0.12 0.01 
Gifted 0.27 0.01 
English learner -0.28 0.02 
Receives special education services -0.25 0.01 
Pretest year is 2012 0.11 0.03 
Pretest year is 2013 0.14 0.04 
Pretest year is 2014 0.19 0.06 
Pretest year is 2015 0.63 0.15 
Imputed for race, age, economic disadvantage, gifted, special education status, and male 0.06 0.20 
Imputed for English learner status -0.86 0.20 
Imputed for baseline score -0.07 0.01 
Constant -0.02 0.04 
Number of enrollments 

Virtual Virginia 964 

Face to face 128,049 

Total 129,013 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 


student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C20. Full model results for the relationship between Virtual Virginia and Standards of Learning score 
outcomes with district fixed effects — Science 


Estimate SE 

Virtual Virginia course -0.30 0.03 
Baseline score 0.61 0.01 
Grade 10 0.18 0.01 
Grade 11 0.20 0.03 
Grade 12 0.28 0.04 
Outcome test is chemistry -0.22 0.01 
Outcome test is earth science 0.20 0.01 
Outcome year is 2016/17 -0.02 0.01 
Student age -0.13 <0.01 
Male -0.01 <0.01 
Black/African American -0.16 0.01 
Hispanic -0.07 0.01 
Asian 0.13 0.01 
Economically disadvantaged -0.13 <0.01 
Gifted 0.30 0.01 
English learner -0.13 0.01 
Receives special education services -0.20 0.01 
Pretest year is 2012 0.19 0.02 
Pretest year is 2013 0.28 0.03 
Pretest year is 2014 0.35 0.04 
Pretest year is 2015 0.40 0.05 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.21 0.38 
Imputed for English learner status -0.54 0.38 
Imputed for baseline score 0.01 0.01 
Constant -0.03 0.04 
Number of enrollments 

Virtual Virginia 1,874 

Face to face 417,792 

Total 419,666 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Standards of Learning proficiency 


The results for the sensitivity analyses based on the dichotomization of SOL exam scores are in Exhibits C21 to 
C23. 


Exhibit C21. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes with district fixed effects - Mathematics 


Estimate SE 

Virtual Virginia course -1.42 0.15 
Baseline score 1.49 0.02 
Grade 9 -0.85 0.08 
Grade 10 -0.84 0.11 
Grade 11 -0.68 0.14 
Grade 12 -0.40 0.18 
Outcome test is algebra Il 0.90 0.06 
Outcome test is geometry 0.24 0.05 
Outcome year is 2016/17 -0.27 0.05 
Student age -0.32 0.01 
Male -0.09 0.01 
Black/African American -0.44 0.02 
Hispanic -0.19 0.03 
Asian 0.40 0.04 
Economically disadvantaged -0.38 0.02 
Gifted 0.92 0.03 
English learner -0.10 0.03 
Receives special education services -0.61 0.02 
Pretest is algebra | 0.95 0.10 
Pretest is geometry 1.09 0.13 
Pretest is mathematics 8 0.41 0.09 
Pretest year is 2012 0.25 0.06 
Pretest year is 2013 0.58 0.10 
Pretest year is 2014 0.82 0.13 
Pretest year is 2015 1.09 0.17 
District percent rural 0.16 0.14 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.48 0.56 
Imputed for English learner status -1.13 0.64 
Imputed for baseline score 0.02 0.02 
Constant -0.56 0.09 
Number of enrollments 

Virtual Virginia 1,765 

Face to face 381,970 

Total 383,735 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C22. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes with district fixed effects — English language arts 


Estimate SE 

Virtual Virginia course -0.27 0.11 
Baseline score 2.18 0.02 
Grade 10 0.92 0.16 
Grade 11 1.82 0.16 
Grade 12 1.86 0.18 
Outcome year is 2016/17 -0.18 0.06 
Student age -0.50 0.02 
Male -0.02 0.02 
Black/African American -0.68 0.03 
Hispanic -0.15 0.04 
Asian -0.09 0.06 
Economically disadvantaged -0.47 0.02 
Gifted 1.40 0.12 
English learner -1.06 0.05 
Receives special education services -1.15 0.03 
Pretest year is 2012 -0.08 0.09 
Pretest year is 2013 0.01 0.13 
Pretest year is 2014 0.11 0.19 
Pretest year is 2015 1.08 0.45 
District percent rural 0.17 0.12 
Imputed for race, age, economic disadvantage, gifted, special education status, and male 0.01 1.26 
Imputed for English learner status -1.04 1.26 
Imputed for baseline score -0.37 0.03 
Constant 0.28 0.15 
Number of enrollments 

Virtual Virginia 964 

Face to face 128,049 

Total 129,013 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C23. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes with district fixed effects — Science 


Estimate SE 

Virtual Virginia course -1.05 0.08 
Baseline score 2.20 0.02 
Grade 10 0.87 0.04 
Grade 11 1.36 0.07 
Grade 12 1.95 0.11 
Outcome test is chemistry -0.11 0.04 
Outcome test is earth science 0.19 0.04 
Outcome year is 2016/17 -0.27 0.03 
Student age -0.49 0.01 
Male 0.04 0.01 
Black/African American -0.67 0.02 
Hispanic -0.32 0.03 
Asian 0.34 0.03 
Economically disadvantaged -0.49 0.02 
Gifted 1.57 0.08 
English learner -0.65 0.04 
Receives special education services -0.96 0.02 
Pretest year is 2012 0.25 0.08 
Pretest year is 2013 0.66 0.10 
Pretest year is 2014 1.05 0.12 
Pretest year is 2015 1.30 0.15 
District percent rural -0.04 0.18 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.79 0.53 
Imputed for English learner status -0.39 0.59 
Imputed for baseline score 0.01 0.04 
Constant -0.12 0.10 
Number of enrollments 

Virtual Virginia 1,874 

Face to face 417,792 

Total 419,666 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Advanced Placement outcomes 
Results of the sensitivity analyses for courses with AP exams follow. 


Advanced Placement test scores 


The results for the sensitivity analyses based on the z-score translation of AP exam scores are in Exhibits C24 to 
C28. 


Exhibit C24. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with district fixed effects — Mathematics 


Estimate SE 

Virtual Virginia course -0.26 0.08 
Baseline score 0.39 0.01 
Grade 10 -0.01 0.10 
Grade 11 -0.04 0.13 
Grade 12 0.08 0.16 
Outcome test is calculus AB -0.15 0.05 
Outcome test is calculus BC -0.33 0.05 
Outcome test is computer science -0.40 0.07 
Outcome year is 2016/17 -0.02 0.02 
Student age -0.15 0.02 
Male 0.13 0.01 
Black/African American -0.24 0.03 
Hispanic -0.15 0.02 
Asian -0.06 0.02 
Economically disadvantaged -0.14 0.02 
Gifted 0.20 0.01 
English learner 0.08 0.05 
Receives special education services 0.09 0.03 
Pretest is algebra | 0.38 0.05 
Pretest is geometry 0.79 0.05 
Pretest year is 2013 0.10 0.03 
Pretest year is 2014 0.33 0.07 
Pretest year is 2015 0.42 0.18 
District percent rural 0.19 0.11 
Imputed for baseline score -0.05 0.07 
Constant -0.64 0.07 
Number of enrollments 

Virtual Virginia 296 

Face to face 29,846 

Total 30,142 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C25. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with district fixed effects — English language arts 


Estimate SE 

Virtual Virginia course 0.02 0.06 
Baseline score 0.39 0.01 
Grade 10 0.51 0.93 
Grade 11 0.66 0.98 
Grade 12 1.05 1.01 
Outcome subject is English language 0.25 0.15 
Outcome year is 2016/17 -0.03 0.02 
Student age -0.11 0.01 
Male -0.04 0.01 
Black/African American -0.31 0.03 
Hispanic -0.12 0.02 
Asian 0.07 0.03 
Economically disadvantaged -0.25 0.02 
Gifted 0.41 0.01 
English learner -0.09 0.05 
Receives special education services -0.05 0.03 
Pretest year is 2013 0.05 0.03 
Pretest year is 2014 0.05 0.24 
Pretest year is 2015 0.05 0.83 
District percent rural -0.34 0.07 
Imputed for baseline score 0.25 0.02 
Constant -0.13 0.07 
Number of enrollments 

Virtual Virginia 291 

Face to face 41,835 

Total 42,126 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C26. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with district fixed effects — Science 


Estimate SE 

Virtual Virginia course -0.22 0.09 
Baseline score 0.46 0.01 
Grade 10 0.37 0.09 
Grade 11 0.32 0.09 
Grade 12 0.45 0.11 
Outcome test is biology 0.18 0.06 
Outcome test is environmental science 0.31 0.06 
Outcome year is 2016/17 0.07 0.03 
Student age -0.12 0.01 
Male 0.18 0.02 
Black/African American -0.35 0.02 
Hispanic -0.17 0.02 
Asian -0.05 0.03 
Economically disadvantaged -0.16 0.02 
Gifted 0.29 0.02 
English learner 0.09 0.05 
Receives special education services <0.01 0.03 
Pretest year is 2013 0.02 0.03 
Pretest year is 2014 0.07 0.06 
Pretest year is 2015 0.08 0.10 
District percent rural 0.27 0.11 
Imputed for baseline score 0.12 0.03 
Constant -0.30 0.07 
Number of enrollments 

Virtual Virginia 437 

Face to face 27,859 

Total 28,296 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C27. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with district fixed effects — History and social science 


Estimate SE 

Virtual Virginia course -0.08 0.05 
Baseline score 0.38 0.01 
Grade 10 0.19 0.06 
Grade 11 0.30 0.07 
Grade 12 0.40 0.08 
Outcome test is art history -0.14 0.09 
Outcome test is European history -0.01 0.04 
Outcome test is comparative government 0.06 0.06 
Outcome test is U.S. government -0.07 0.04 
Outcome test is human geography 0.13 0.04 
Outcome test is macroeconomics -0.39 0.09 
Outcome test is microeconomics -0.29 0.07 
Outcome test is psychology 0.08 0.04 
Outcome test is U.S. history <0.01 0.05 
Outcome year is 2016/17 0.03 0.02 
Student age -0.10 0.01 
Male 0.13 0.01 
Black/African American -0.24 0.02 
Hispanic -0.13 0.01 
Asian 0.05 0.02 
Economically disadvantaged -0.18 0.01 
Gifted 0.36 0.01 
English learner -0.08 0.03 
Receives special education services -0.06 0.02 
Pretest year is 2013 -0.03 0.02 
Pretest year is 2014 -0.02 0.04 
Pretest year is 2015 -0.03 0.07 
District percent rural 0.46 0.29 
Imputed for baseline score 0.19 0.01 
Constant -0.08 0.13 
Number of enrollments 

Virtual Virginia 2,143 

Face to face 106,974 

Total 109,117 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C28. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with district fixed effects — Languages 


Estimate SE 

Virtual Virginia course -0.02 0.19 
Baseline score 0.27 0.02 
Grade 11 -0.20 0.19 
Grade 12 -0.25 0.17 
Outcome test is Chinese -0.24 0.18 
Outcome test is French 0.07 0.09 
Outcome test is Latin -0.03 0.13 
Outcome year is 2016/17 -0.11 0.18 
Student age -0.17 0.04 
Male -0.02 0.03 
Black/African American -0.26 0.08 
Hispanic 0.57 0.06 
Asian 0.05 0.09 
Economically disadvantaged -0.01 0.07 
Gifted 0.19 0.04 
English learner 0.36 0.07 
Receives special education services -0.30 0.08 
Pretest year is 2013 0.06 0.15 
Pretest year is 2014 0.05 0.17 
District percent rural -0.29 0.22 
Imputed for baseline score 0.34 0.05 
Constant -0.19 0.14 
Number of enrollments 

Virtual Virginia 136 

Face to face 3,417 

Total 3,553 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Advanced Placement proficiency 


The results for the sensitivity analyses based on the dichotomization of AP exam scores are in exhibits C29 to C33. 


Exhibit C29. Full model results for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with district fixed effects - Mathematics 


Estimate SE 

Virtual Virginia course -0.68 0.20 
Baseline score 0.88 0.03 
Grade 10 -0.46 1.01 
Grade 11 -0.48 1.12 
Grade 12 -0.15 1.17 
Outcome test is calculus AB 0.11 0.11 
Outcome test is calculus BC 0.63 0.12 
Outcome test is computer science -0.09 0.17 
Outcome year is 2016/17 -0.20 0.05 
Student age -0.33 0.05 
Male 0.28 0.03 
Black/African American -0.59 0.08 
Hispanic -0.34 0.06 
Asian -0.13 0.06 
Economically disadvantaged -0.37 0.06 
Gifted 0.47 0.03 
English learner 0.19 0.14 
Receives special education services 0.31 0.10 
Pretest is algebra | 1.06 0.15 
Pretest is geometry 1.92 0.15 
Pretest year is 2013 0.30 0.09 
Pretest year is 2014 1.03 0.21 
Pretest year is 2015 0.58 1.55 
District percent rural 0.62 0.28 
Imputed for baseline score -0.05 0.18 
Constant -1.31 0.18 
Number of enrollments 

Virtual Virginia 294 

Face to face 29,819 

Total 30,113 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C30. Full model results for the relationship between Virtual Virginia and Advanced Placement 
Proficiency outcome with district fixed effects — English language arts 


Estimate SE 

Virtual Virginia course 0.19 0.15 
Baseline score 0.99 0.02 
Grade 10 -0.62 2.64 
Grade 11 -0.11 2.73 
Grade 12 0.53 2.78 
Outcome subject is English language 0.36 0.29 
Outcome year is 2016/17 -0.14 0.06 
Student age -0.28 0.04 
Male -0.06 0.03 
Black/African American -0.69 0.07 
Hispanic -0.22 0.05 
Asian 0.21 0.07 
Economically disadvantaged -0.62 0.06 
Gifted 0.90 0.04 
English learner -0.49 0.13 
Receives special education services -0.18 0.09 
Pretest year is 2013 0.25 0.08 
Pretest year is 2014 -0.35 0.56 
Pretest year is 2015 -1.21 2.31 
District percent rural -0.70 0.14 
Imputed for baseline score 0.55 0.05 
Constant 0.48 0.15 
Number of enrollments 

Virtual Virginia 291 

Face to face 41,835 

Total 42,126 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C31. Full model results for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with district fixed effects — Science 


Estimate SE 

Virtual Virginia course -0.68 0.20 
Baseline score 1.16 0.03 
Grade 10 0.80 0.20 
Grade 11 0.89 0.27 
Grade 12 1.18 0.30 
Outcome test is biology 2.21 0.13 
Outcome test is environmental science 1.40 0.15 
Outcome year is 2016/17 0.44 0.07 
Student age -0.30 0.04 
Male 0.48 0.05 
Black/African American -1.05 0.07 
Hispanic -0.42 0.07 
Asian -0.15 0.05 
Economically disadvantaged -0.48 0.06 
Gifted 0.70 0.04 
English learner 0.18 0.15 
Receives special education services -0.05 0.08 
Pretest year is 2013 -0.05 0.08 
Pretest year is 2014 0.08 0.19 
Pretest year is 2015 0.11 0.31 
District percent rural 0.69 0.28 
Imputed for baseline score 0.36 0.07 
Constant -1.41 0.18 
Number of enrollments 

Virtual Virginia 455 

Face to face 28,013 

Total 28,468 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C32. Full model results for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with district fixed effects — History and social science 


Estimate SE 

Virtual Virginia course -0.24 0.11 
Baseline score 0.94 0.02 
Grade 10 0.44 0.14 
Grade 11 0.70 0.18 
Grade 12 0.81 0.20 
Outcome test is art history -0.26 0.25 
Outcome test is European history -0.95 0.10 
Outcome test is comparative government -0.62 0.14 
Outcome test is U.S. government -0.41 0.10 
Outcome test is human geography -0.12 0.09 
Outcome test is macroeconomics -0.64 0.20 
Outcome test is microeconomics -0.37 0.17 
Outcome test is psychology 0.27 0.10 
Outcome test is U.S. history -0.49 0.11 
Outcome year is 2016/17 0.16 0.04 
Student age -0.23 0.02 
Male 0.35 0.03 
Black/African American -0.56 0.04 
Hispanic -0.32 0.03 
Asian 0.07 0.05 
Economically disadvantaged -0.37 0.03 
Gifted 0.79 0.03 
English learner -0.13 0.07 
Receives special education services -0.13 0.05 
Pretest year is 2013 -0.23 0.05 
Pretest year is 2014 -0.13 0.10 
Pretest year is 2015 -0.27 0.18 
District percent rural 1.15 0.60 
Imputed for baseline score 0.40 0.04 
Constant 0.69 0.27 
Number of enrollments 

Virtual Virginia 2,143 

Face to face 106,974 

Total 109,117 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C33. Full model results for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with district fixed effects — Languages 


Estimate SE 

Virtual Virginia course -0.33 0.48 
Baseline score 0.67 0.08 
Grade 10 13.04 1.48 
Grade 11 -0.26 0.98 
Grade 12 0.27 0.52 
Outcome test is Chinese -0.28 0.59 
Outcome test is French -1.22 0.23 
Outcome test is Latin -2.24 0.29 
Outcome year is 2016/17 -0.90 0.68 
Student age -0.39 0.14 
Male -0.20 0.10 
Black/African American -0.85 0.21 
Hispanic 1.20 0.22 
Asian 0.30 0.21 
Economically disadvantaged -0.09 0.19 
Gifted 0.42 0.16 
English learner 1.44 0.39 
Receives special education services -0.52 0.25 
Pretest year is 2013 0.65 0.81 
Pretest year is 2014 -11.71 1.45 
District percent rural -0.68 0.72 
Imputed for baseline score 0.63 0.23 
Constant 3.30 0.44 
Number of enrollments 

Virtual Virginia 135 

Face to face 3,404 

Total 3,539 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Alternative specifications of the demographic variables 

As discussed in the “Data intake and cleaning” section of appendix A, the raw data the Virginia Department of 
Education supplied included duplicate records for students, with variation in the baseline achievement and 
demographic variables. This section presents results of alternative approaches to eliminating those duplicates. 


In the models presented in appendix B, the study team used “ever” black, Hispanic, economically disadvantaged, 
gifted, English learner, received special education services, and male; that is, we created a variable that reflected 
if students ever appeared with a “yes” for the variable. In this sensitivity analysis, we use the inverse of this 
variable (that is, if a student was ever recorded as not economically disadvantaged in any record in the year, he 
or she was recorded as not economically disadvantaged for the entire year). For age, average age was used in the 
main analysis, whereas in this sensitivity analysis youngest age was used. Finally, for student race, division, school, 
and grade level, the study team selected the earliest instance in the school year for the main analysis and used 
the second instance for this analysis. 


These alternative specifications of the control variables did not substantially change the results. 
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Standards of Learning outcomes 
Results of the sensitivity analyses for courses with SOL exams follow. 


Standards of Learning test scores 


The results for the sensitivity analyses based on the z-score translation of continuous SOL exam scores are in 
exhibits C34 to C36. 


Exhibit C34. Full model results for the relationship between Virtual Virginia and Standards of Learning score 
outcomes with alternate control variable specifications - Mathematics 


Estimate SE 

Virtual Virginia course -0.39 0.04 
Baseline score 0.49 0.01 
Grade 9 -0.38 0.03 
Grade 10 -0.46 0.04 
Grade 11 -0.54 0.06 
Grade 12 -0.55 0.07 
Outcome test is algebra Il 0.02 0.02 
Outcome test is geometry 0.14 0.02 
Outcome year is 2016/17 0.04 0.02 
Student age -0.10 <0.01 
Male -0.02 <0.01 
Black/African American -0.16 0.01 
Hispanic -0.06 0.01 
Asian 0.19 0.01 
Economically disadvantaged -0.11 0.01 
Gifted 0.17 0.27 
English learner -0.02 0.01 
Receives special education services -0.12 0.01 
Pretest is algebra | 0.59 0.03 
Pretest is geometry 0.98 0.04 
Pretest is mathematics 8 0.14 0.03 
Pretest year is 2012 0.08 0.02 
Pretest year is 2013 0.19 0.04 
Pretest year is 2014 0.30 0.05 
Pretest year is 2015 0.31 0.07 
District percent rural -0.10 0.02 
Imputed for race, age, economic disadvantage, gifted, special education status, and male 0.03 0.19 
Imputed for English learner status -0.89 0.20 
Imputed for baseline score 0.11 0.01 
Constant -0.31 0.02 
Number of enrollments 

Virtual Virginia 1,765 

Face to face 381,970 

Total 383,735 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C35. Full model results for the relationship between Virtual Virginia and Standards of Learning score 
outcomes with alternate control variable specifications — English language arts 


Estimate SE 

Virtual Virginia course -0.04 0.03 
Baseline score 0.61 <0.01 
Grade 10 0.23 0.07 
Grade 11 0.44 0.07 
Grade 12 0.38 0.08 
Outcome year is 2016/17 0.02 0.02 
Student age -0.16 0.01 
Male <0.01 <0.01 
Black/African American -0.20 0.01 
Hispanic -0.05 0.01 
Asian 0.02 0.02 
Economically disadvantaged -0.13 0.01 
Gifted 0.56 0.24 
English learner -0.24 0.02 
Receives special education services -0.23 0.01 
Pretest year is 2012 0.08 0.03 
Pretest year is 2013 0.08 0.04 
Pretest year is 2014 0.09 0.06 
Pretest year is 2015 0.51 0.16 
District percent rural -0.06 0.02 
Imputed for race, age, economic disadvantage, gifted, special education status, and male 0.07 0.22 
Imputed for English learner status -0.82 0.24 
Imputed for baseline score -0.09 0.01 
Constant 0.03 0.04 
Number of enrollments 

Virtual Virginia 964 

Face to face 128,049 

Total 129,013 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C36. Full model results for the relationship between Virtual Virginia and Standards of Learning score 
outcomes with alternate control variable specifications — Science 


Estimate SE 

Virtual Virginia course -0.31 0.03 
Baseline score 0.65 0.01 
Grade 10 0.17 0.01 
Grade 11 0.17 0.03 
Grade 12 0.26 0.04 
Outcome test is chemistry -0.18 0.01 
Outcome test is earth science 0.17 0.01 
Outcome year is 2016/17 -0.04 0.01 
Student age -0.13 <0.01 
Male -0.02 <0.01 
Black/African American -0.19 0.01 
Hispanic -0.08 0.01 
Asian 0.15 0.01 
Economically disadvantaged -0.12 0.01 
Gifted 0.41 0.27 
English learner -0.12 0.01 
Receives special education services -0.19 0.01 
Pretest year is 2012 0.19 0.02 
Pretest year is 2013 0.30 0.03 
Pretest year is 2014 0.39 0.04 
Pretest year is 2015 0.45 0.05 
District percent rural -0.07 0.01 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.21 0.40 
Imputed for English learner status -0.49 0.40 
Imputed for baseline score -0.01 0.02 
Constant -0.05 0.01 
Number of enrollments 

Virtual Virginia 1,874 

Face to face 417,792 

Total 419,666 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Standards of Learning proficiency 


The results for the sensitivity analyses based on the dichotomization of SOL exam scores are in exhibits C37 to 
C39. 


Exhibit C37. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes with alternate control variable specifications - Mathematics 


Estimate SE 

Virtual Virginia course -1.40 0.15 
Baseline score 1.56 0.02 
Grade 9 -0.91 0.10 
Grade 10 -0.89 0.12 
Grade 11 -0.74 0.15 
Grade 12 -0.43 0.19 
Outcome test is algebra Il 0.98 0.06 
Outcome test is geometry 0.28 0.05 
Outcome year is 2016/17 -0.26 0.05 
Student age -0.34 0.02 
Male -0.09 0.01 
Black/African American -0.57 0.04 
Hispanic -0.22 0.04 
Asian 0.45 0.04 
Economically disadvantaged -0.37 0.02 
Gifted 0.73 0.89 
English learner -0.15 0.04 
Receives special education services -0.58 0.03 
Pretest is algebra | 1.07 0.11 
Pretest is geometry 1.36 0.15 
Pretest is mathematics 8 0.57 0.10 
Pretest year is 2012 0.25 0.06 
Pretest year is 2013 0.57 0.09 
Pretest year is 2014 0.81 0.13 
Pretest year is 2015 1.10 0.17 
District percent rural -0.26 0.06 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.39 0.58 
Imputed for English learner status -1.38 0.64 
Imputed for baseline score -0.03 0.02 
Constant -0.75 0.07 
Number of enrollments 

Virtual Virginia 1,765 

Face to face 381,970 

Total 383,735 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C38. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes with alternate control variable specifications — English language arts 


Estimate SE 

Virtual Virginia course -0.24 0.12 
Baseline score 2.30 0.02 
Grade 10 0.83 0.19 
Grade 11 1.66 0.20 
Grade 12 1.63 0.22 
Outcome year is 2016/17 -0.08 0.06 
Student age -0.54 0.02 
Male -0.01 0.02 
Black/African American -0.66 0.04 
Hispanic -0.02 0.04 
Asian 0.22 0.06 
Economically disadvantaged -0.50 0.03 
Gifted 0.01 0.62 
English learner -0.88 0.05 
Receives special education services -1.07 0.03 
Pretest year is 2012 -0.24 0.08 
Pretest year is 2013 -0.25 0.12 
Pretest year is 2014 -0.36 0.19 
Pretest year is 2015 0.61 0.50 
District percent rural -0.18 0.06 
Imputed for race, age, economic disadvantage, gifted, special education status, and male <0.01 1.20 
Imputed for English learner status -1.19 1.23 
Imputed for baseline score -0.37 0.03 
Constant 0.28 0.15 
Number of enrollments 

Virtual Virginia 964 

Face to face 128,049 

Total 129,013 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C39. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes with alternate control variable specifications — Science 


Estimate SE 

Virtual Virginia course -1.06 0.09 
Baseline score 2.33 0.02 
Grade 10 0.89 0.05 
Grade 11 1.34 0.09 
Grade 12 1.94 0.13 
Outcome test is chemistry 0.03 0.04 
Outcome test is earth science 0.13 0.04 
Outcome year is 2016/17 -0.32 0.04 
Student age -0.51 0.01 
Male 0.03 0.01 
Black/African American -0.75 0.03 
Hispanic -0.31 0.03 
Asian 0.48 0.04 
Economically disadvantaged -0.49 0.02 
Gifted 0.57 0.82 
English learner -0.63 0.04 
Receives special education services -0.93 0.02 
Pretest year is 2012 0.18 0.08 
Pretest year is 2013 0.60 0.11 
Pretest year is 2014 1.02 0.14 
Pretest year is 2015 1.30 0.17 
District percent rural -0.04 0.04 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.69 0.53 
Imputed for English learner status -0.56 0.58 
Imputed for baseline score -0.06 0.04 
Constant -0.10 0.03 
Number of enrollments 

Virtual Virginia 1,874 

Face to face 417,792 

Total 419,666 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Cc-40 


Advanced Placement outcomes 
Results of the sensitivity analyses for courses with AP exams follow. 


Advanced Placement test scores 


The results for the sensitivity analyses based on the z-score translation of continuous AP exam scores are in 
exhibits C40 to C44. 


Exhibit C40. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with alternate control variable specifications - Mathematics 


Estimate SE 

Virtual Virginia course -0.33 0.07 
Baseline score 0.42 0.01 
Grade 10 -0.06 0.11 
Grade 11 -0.15 0.14 
Grade 12 -0.04 0.16 
Outcome test is calculus AB -0.12 0.05 
Outcome test is calculus BC -0.30 0.05 
Outcome test is computer science -0.33 0.07 
Outcome year is 2016/17 -0.03 0.02 
Student age -0.14 0.02 
Male 0.13 0.01 
Black/African American -0.33 0.03 
Hispanic -0.17 0.03 
Asian 0.01 0.03 
Economically disadvantaged -0.22 0.03 
Gifted N/A 

English learner 0.05 0.05 
Receives special education services 0.13 0.04 
Pretest is algebra | 0.48 0.04 
Pretest is geometry 0.93 0.05 
Pretest year is 2013 0.08 0.04 
Pretest year is 2014 0.32 0.07 
Pretest year is 2015 0.40 0.20 
District percent rural -0.12 0.06 
Imputed for baseline score 0.02 0.07 
Constant -0.48 0.06 
Number of enrollments 

Virtual Virginia 296 

Face to face 29,846 

Total 30,142 


N/A is not applicable. 

Notes: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C41. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with alternate control variable specifications — English language arts 


Estimate SE 

Virtual Virginia course -0.01 0.07 
Baseline score 0.47 0.01 
Grade 10 1.60 0.23 
Grade 11 1.69 0.19 
Grade 12 2.15 0.20 
Outcome subject is English language 0.33 0.14 
Outcome year is 2016/17 -0.04 0.03 
Student age -0.11 0.01 
Male -0.01 0.01 
Black/African American -0.38 0.03 
Hispanic -0.13 0.02 
Asian 0.14 0.03 
Economically disadvantaged -0.31 0.02 
Gifted 0.95 0.18 
English learner -0.14 0.05 
Receives special education services -0.01 0.03 
Pretest year is 2013 0.06 0.03 
Pretest year is 2014 -0.04 0.19 
Pretest year is 2015 1.18 0.54 
District percent rural -0.21 0.05 
Imputed for baseline score 0.14 0.02 
Constant -0.18 0.06 
Number of enrollments 

Virtual Virginia 291 

Face to face 41,835 

Total 42,126 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C42. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with alternate control variable specifications — Science 


Estimate SE 

Virtual Virginia course -0.22 0.07 
Baseline score 0.52 0.01 
Grade 10 0.41 0.12 
Grade 11 0.41 0.11 
Grade 12 0.53 0.13 
Outcome test is biology 0.17 0.05 
Outcome test is environmental science 0.27 0.06 
Outcome year is 2016/17 0.05 0.03 
Student age -0.11 0.01 
Male 0.17 0.02 
Black/African American -0.42 0.03 
Hispanic -0.20 0.03 
Asian -0.01 0.03 
Economically disadvantaged -0.23 0.02 
Gifted N/A 

English learner 0.09 0.05 
Receives special education services 0.04 0.03 
Pretest year is 2013 0.02 0.03 
Pretest year is 2014 0.08 0.06 
Pretest year is 2015 0.08 0.10 
District percent rural -0.21 0.06 
Imputed for baseline score 0.01 0.04 
Constant -0.14 0.04 
Number of enrollments 

Virtual Virginia 437 

Face to face 27,859 

Total 28,296 


N/A is not applicable. 

In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each student's 
AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A negative 
number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a student was 
above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C43. Full model output for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with alternate control variable specifications — History and social science 


Estimate SE 

Virtual Virginia course -0.28 0.05 
Baseline score 0.46 0.01 
Grade 10 0.27 0.09 
Grade 11 0.40 0.10 
Grade 12 0.46 0.11 
Outcome test is art history -0.03 0.10 
Outcome test is European history -0.01 0.06 
Outcome test is comparative government 0.02 0.07 
Outcome test is U.S. government -0.03 0.06 
Outcome test is human geography 0.15 0.05 
Outcome test is macroeconomics -0.29 0.10 
Outcome test is microeconomics -0.19 0.09 
Outcome test is psychology 0.09 0.06 
Outcome test is U.S. history -0.04 0.06 
Outcome year is 2016/17 0.01 0.02 
Student age -0.09 0.01 
Male 0.14 0.01 
Black/African American -0.35 0.03 
Hispanic -0.16 0.02 
Asian 0.14 0.03 
Economically disadvantaged -0.28 0.02 
Gifted 0.85 0.59 
English learner -0.06 0.03 
Receives special education services <0.01 0.02 
Pretest year is 2013 -0.02 0.02 
Pretest year is 2014 -0.02 0.04 
Pretest year is 2015 -0.04 0.08 
District percent rural -0.15 0.06 
Imputed for baseline score 0.10 0.02 
Constant 0.01 0.04 
Number of enrollments 

Virtual Virginia 2,143 

Face to face 106,974 

Total 109,117 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C44. Full model output for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with alternate control variable specifications — Languages 


Estimate SE 

Virtual Virginia course -0.24 0.18 
Baseline score 0.30 0.02 
Grade 11 -0.20 0.20 
Grade 12 -0.23 0.21 
Outcome test is Chinese -0.19 0.16 
Outcome test is French 0.09 0.09 
Outcome test is Latin -0.01 0.13 
Outcome year is 2016/17 -0.25 0.19 
Student age -0.16 0.04 
Male -0.02 0.04 
Black/African American -0.27 0.08 
Hispanic 0.58 0.06 
Asian 0.12 0.12 
Economically disadvantaged 0.02 0.05 
Gifted 0.92 0.31 
English learner 0.34 0.07 
Receives special education services -0.22 0.08 
Pretest year is 2013 0.01 0.16 
Pretest year is 2014 <0.01 0.20 
District percent rural -0.20 0.11 
Imputed for baseline score 0.29 0.05 
Constant -0.03 0.11 
Number of enrollments 

Virtual Virginia 136 

Face to face 3,417 

Total 3,553 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


C45 


Advanced Placement proficiency 


The results for the sensitivity analyses based on the dichotomization of AP exam scores are in exhibits C45 to C49. 


Exhibit C45. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcome with alternate control variable specifications — Mathematics 


Estimate SE 

Virtual Virginia course -0.76 0.18 
Baseline score 0.92 0.03 
Grade 10 -0.71 1.03 
Grade 11 -0.86 1.13 
Grade 12 -0.60 1.18 
Outcome test is calculus AB 0.17 0.10 
Outcome test is calculus BC 0.72 0.11 
Outcome test is computer science 0.05 0.16 
Outcome year is 2016/17 -0.20 0.05 
Student age -0.28 0.05 
Male 0.27 0.03 
Black/African American -0.75 0.09 
Hispanic -0.39 0.06 
Asian <0.01 0.06 
Economically disadvantaged -0.54 0.07 
Gifted N/A 

English learner 0.09 0.14 
Receives special education services 0.39 0.11 
Pretest is algebra | 1.24 0.14 
Pretest is geometry 2.17 0.16 
Pretest year is 2013 0.24 0.09 
Pretest year is 2014 0.95 0.20 
Pretest year is 2015 0.36 1.61 
District percent rural -0.22 0.12 
Imputed for baseline score 0.09 0.17 
Constant -0.87 0.17 
Number of enrollments 

Virtual Virginia 296 

Face to face 29,846 

Total 30,142 


N/A is not applicable. 

Notes: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


C-46 


Exhibit C46. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with alternate control variable specifications — English language arts 


Estimate SE 

Virtual Virginia course 0.10 0.16 
Baseline score 1.12 0.02 
Grade 10 14.19 0.72 
Grade 11 14.65 0.73 
Grade 12 15.42 0.61 
Outcome subject is English language 0.55 0.28 
Outcome year is 2016/17 -0.15 0.06 
Student age -0.25 0.04 
Male 0.01 0.03 
Black/African American -0.81 0.07 
Hispanic -0.25 0.06 
Asian 0.36 0.08 
Economically disadvantaged -0.72 0.06 
Gifted N/A 

English learner -0.65 0.15 
Receives special education services -0.09 0.09 
Pretest year is 2013 0.24 0.07 
Pretest year is 2014 -0.34 0.52 
Pretest year is 2015 13.73 1.15 
District percent rural -0.40 0.11 
Imputed for baseline score 0.32 0.06 
Constant 0.23 0.13 
Number of enrollments 

Virtual Virginia 291 

Face to face 41,834 

Total 42,125 


N/A is not applicable. 

Notes: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C47. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with alternate control variable specifications — Science 


Estimate SE 

Virtual Virginia course -0.62 0.17 
Baseline score 1.25 0.03 
Grade 10 0.84 0.24 
Grade 11 0.97 0.33 
Grade 12 1.22 0.35 
Outcome test is biology 2.10 0.13 
Outcome test is environmental science 1.27 0.15 
Outcome year is 2016/17 0.39 0.07 
Student age -0.27 0.04 
Male 0.43 0.04 
Black/African American -1.13 0.09 
Hispanic -0.49 0.08 
Asian -0.07 0.05 
Economically disadvantaged -0.64 0.07 
Gifted N/A 

English learner 0.11 0.16 
Receives special education services 0.08 0.08 
Pretest year is 2013 -0.05 0.08 
Pretest year is 2014 0.09 0.18 
Pretest year is 2015 0.06 0.27 
District percent rural -0.40 0.12 
Imputed for baseline score 0.05 0.08 
Constant -0.97 0.11 
Number of enrollments 

Virtual Virginia 456 

Face to face 28,022 

Total 28,478 


N/A is not applicable. 

Notes: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C48. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with alternate control variable specifications — History and social science 


Estimate SE 

Virtual Virginia course -0.67 0.10 
Baseline score 1.05 0.02 
Grade 10 0.57 0.21 
Grade 11 0.87 0.24 
Grade 12 0.86 0.26 
Outcome test is art history 0.05 0.26 
Outcome test is European history -0.86 0.14 
Outcome test is comparative government -0.64 0.16 
Outcome test is U.S. government -0.30 0.13 
Outcome test is human geography -0.07 0.12 
Outcome test is macroeconomics -0.35 0.22 
Outcome test is microeconomics -0.12 0.20 
Outcome test is psychology 0.27 0.13 
Outcome test is U.S. history -0.55 0.13 
Outcome year is 2016/17 0.11 0.04 
Student age -0.21 0.02 
Male 0.36 0.03 
Black/African American -0.75 0.07 
Hispanic -0.36 0.05 
Asian 0.26 0.07 
Economically disadvantaged -0.57 0.05 
Gifted N/A 

English learner -0.13 0.08 
Receives special education services <0.01 0.05 
Pretest year is 2013 -0.21 0.05 
Pretest year is 2014 -0.13 0.10 
Pretest year is 2015 -0.33 0.20 
District percent rural -0.26 0.11 
Imputed for baseline score 0.22 0.04 
Constant 0.85 0.09 
Number of enrollments 

Virtual Virginia 2,143 

Face to face 106,973 

Total 109,116 


N/A is not applicable. 

Notes: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C49. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with alternate control variable specifications — Languages 


Estimate SE 

Virtual Virginia course -0.51 0.41 
Baseline score 0.69 0.09 
Grade 10 13.61 1.23 
Grade 11 0.13 0.92 
Grade 12 0.45 0.53 
Outcome test is Chinese -0.33 0.58 
Outcome test is French -1.24 0.23 
Outcome test is Latin -2.10 0.27 
Outcome year is 2016/17 -0.93 0.60 
Student age -0.36 0.14 
Male -0.20 0.11 
Black/African American -0.88 0.22 
Hispanic 1.13 0.22 
Asian 0.41 0.23 
Economically disadvantaged -0.25 0.19 
Gifted N/A 

English learner 1.39 0.41 
Receives special education services -0.36 0.29 
Pretest year is 2013 0.24 0.70 
Pretest year is 2014 -12.35 1.19 
District percent rural -0.35 0.25 
Imputed for baseline score 0.51 0.20 
Constant 3.40 0.26 
Number of enrollments 

Virtual Virginia 136 

Face to face 3,416 

Total 3,552 


N/A is not applicable. 

Notes: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Full model results with interaction terms for Standards of Learning outcomes 


The coefficients (estimates) and standard errors for all variables included in the models used to estimate the 
relationship between taking a course online through VVA and SOL exam outcomes are in exhibits C50 to C53, 
including an interaction term for the student’s grade level and the course the student took. The first row of each 
table displays the coefficient of interest: the relationship between VVA and the test outcome. Subsequent rows 
display each of the control variables in the model and the number of enrollments in that model. These models 
counted number of enrollments rather than number of students because a student may have been enrolled in 
more than one course in a subject area. (Exhibits C2 to C17 present the results of a sensitivity test aggregating 
results to the student level.) No substantive differences were found in the estimates of the relationship between 
VVA and test score outcomes. 


Standards of Learning test scores 


This section includes the linear regression results estimating the relationship between Virtual Virginia and SOL z- 
scores, including an interaction term for the student’s grade level and the course the student took. 


Exhibit C50. Full model results for the relationship between Virtual Virginia and Standards of Learning score 
outcomes with a grade-by-course interaction term — Mathematics 


Estimate SE 

Virtual Virginia course -0.46 0.04 
Baseline score 0.47 0.01 
Grade 9 -0.18 0.03 
Grade 10 -0.20 0.05 
Grade 11 -0.11 0.06 
Grade 12 0.04 0.10 
Outcome test is algebra Il -0.08 0.02 
Outcome test is geometry 0.04 0.01 
Outcome year is 2016/17 0.02 0.02 
Student age -0.10 <0.01 
Male -0.01 <0.01 
Black/African American -0.14 0.01 
Hispanic -0.05 0.01 
Asian 0.19 0.01 
Economically disadvantaged -0.12 0.01 
Gifted 0.27 0.01 
English learner 0.00 0.01 
Receives special education services -0.13 0.01 
Pretest is algebra | 0.42 0.03 
Pretest is geometry 0.69 0.07 
Pretest is mathematics 8 0.09 0.03 
Pretest year is 2012 0.10 0.02 
Pretest year is 2013 0.21 0.04 
Pretest year is 2014 0.33 0.06 
Pretest year is 2015 0.36 0.07 
District percent rural -0.09 0.02 
Imputed for race, age, economic disadvantage, gifted, special education status, and male 0.01 0.16 
Imputed for English learner status -1.00 0.16 
Imputed for baseline score 0.09 0.01 
Grade 9* algebra II interaction 0.62 0.08 
Grade 9*geometry interaction 0.46 0.05 
Grade 10*algebra II interaction 0.51 0.05 
Grade 10*geometry interaction 0.28 0.04 
Grade 11*algebra II interaction 0.22 0.04 
Grade 11*geometry interaction 0.12 0.04 
Constant -0.21 0.02 
Number of enrollments 

Virtual Virginia 1,765 

Face to face 381,970 

Total 383,735 
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Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1.A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 


Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
SSS SSS Se 


Exhibit C51. Full model results for the relationship between Virtual Virginia and Standards of Learning score 
outcomes with a grade-by-course interaction term — Science 


Estimate SE 

Virtual Virginia course -0.34 0.03 
Baseline score 0.61 0.01 
Grade 10 0.14 0.01 
Grade 11 0.22 0.03 
Grade 12 0.34 0.04 
Outcome test is chemistry -0.16 0.02 
Outcome test is earth science 0.21 0.01 
Outcome year is 2016/17 -0.03 0.01 
Student age -0.13 <0.01 
Male -0.01 <0.01 
Black/African American -0.19 0.01 
Hispanic -0.07 0.01 
Asian 0.13 0.01 
Economically disadvantaged -0.13 0.01 
Gifted 0.29 0.01 
English learner -0.12 0.01 
Receives special education services -0.19 0.01 
Pretest year is 2012 0.17 0.02 
Pretest year is 2013 0.27 0.03 
Pretest year is 2014 0.36 0.04 
Pretest year is 2015 0.41 0.05 
District percent rural -0.07 0.01 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.20 0.39 
Imputed for English learner status -0.55 0.40 
Imputed for baseline score -0.01 0.01 
Grade 10*chemistry interaction -0.08 0.03 
Grade 10*earth science interaction 0.03 0.02 
Grade 11*chemistry interaction -0.19 0.04 
Grade 11*earth science interaction 0.16 0.03 
Grade 12*chemistry interaction -0.27 0.04 
Grade 12*earth science interaction 0.17 0.03 
Constant -0.03 0.01 
Number of enrollments 

Virtual Virginia 1,874 

Face to face 417,792 

Total 419,666 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from SOL scaled scores. Z-scores were created by taking each 
student's SOL scaled score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of 0 and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Standards of Learning proficiency 


The logistic regression results estimating the relationship between VVA and proficiency on the SOL exam (a score 
of 400 or higher) are in exhibits C52 and C53, including an interaction term for the student’s grade level and the 
course the student took. 
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Exhibit C52. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes with a grade-by-course interaction term — Mathematics 


Estimate Se 

Virtual Virginia course -1.47 0.14 
Baseline score 1.51 0.02 
Grade 9 -0.49 0.11 
Grade 10 -0.44 0.13 
Grade 11 0.09 0.16 
Grade 12 0.75 0.23 
Outcome test is algebra Il 0.68 0.08 
Outcome test is geometry 0.03 0.05 
Outcome year is 2016/17 -0.28 0.05 
Student age -0.33 0.01 
Male -0.08 0.01 
Black/African American -0.51 0.04 
Hispanic -0.18 0.04 
Asian 0.46 0.04 
Economically disadvantaged -0.42 0.02 
Gifted 0.97 0.04 
English learner -0.09 0.04 
Receives special education services -0.58 0.03 
Pretest is algebra | 0.79 0.11 
Pretest is geometry 0.93 0.23 
Pretest is math8 0.44 0.10 
Pretest year is 2012 0.28 0.06 
Pretest year is 2013 0.62 0.10 
Pretest year is 2014 0.88 0.15 
Pretest year is 2015 1.19 0.19 
District percent rural -0.25 0.06 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.56 0.50 
Imputed for English learner status -1.25 0.55 
Imputed for baseline score -0.06 0.02 
Grade 9* algebra II interaction 0.92 0.27 
Grade 9*geometry interaction 0.97 0.16 
Grade 10*algebra II interaction 1.05 0.14 
Grade 10*geometry interaction 0.91 0.15 
Grade 11*algebra II interaction 0.48 0.14 
Grade 11*geometry interaction 0.35 0.15 
Constant -0.46 0.07 
Number of enrollments 

Virtual Virginia 1,765 

Face to face 381,970 

Total 383,735 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C53. Full model results for the relationship between Virtual Virginia and Standards of Learning 
proficiency outcomes with a grade-by-course interaction term — Science 


Estimate SE 

Virtual Virginia course -1.17 0.08 
Baseline score 2.21 0.02 
Grade 10 0.74 0.06 
Grade 11 1.20 0.09 
Grade 12 2.01 0.13 
Outcome test is chemistry 0.16 0.14 
Outcome test is earth science 0.29 0.04 
Outcome year is 2016/17 -0.30 0.04 
Student age -0.48 0.01 
Male 0.04 0.01 
Black/African American -0.71 0.03 
Hispanic -0.29 0.03 
Asian 0.40 0.04 
Economically disadvantaged -0.52 0.02 
Gifted 1.52 0.08 
English learner -0.59 0.04 
Receives special education services -0.90 0.02 
Pretest year is 2012 0.17 0.08 
Pretest year is 2013 0.59 0.11 
Pretest year is 2014 1.00 0.13 
Pretest year is 2015 1.28 0.16 
District percent rural -0.04 0.04 
Imputed for race, age, economic disadvantage, gifted, special education status, and male -0.50 0.50 
Imputed for English learner status -0.80 0.54 
Imputed for baseline score -0.08 0.04 
Grade 10*chemistry interaction -0.08 0.32 
Grade 10* earth science interaction 0.19 0.07 
Grade 11*chemistry interaction -0.26 0.32 
Grade 11*earth science interaction 0.77 0.09 
Grade 12*chemistry interaction -0.68 0.33 
Grade 12*earth science interaction 0.91 0.12 
Constant 0.15 0.03 
Number of enrollments 

Virtual Virginia 1,874 

Face to face 417,792 

Total 419,666 


Note: In this analysis, SOL proficiency was used as an outcome. SOL scaled scores range from 0 to 600, where a score of 0-399 means a student did not pass 
the test. A score of 400-499 means a student passed a test and was classified as proficient, and a score of 500-600 means a student passed a test and was 
classified as advanced. The outcome used in these analyses is dichotomous, with scores of 400 or above scored as 1 and scores of 399 and below scored as 
0. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Full model results with interaction terms for Advanced Placement outcomes 

This section shows the coefficients (estimates) and standard errors for all variables included in the models used 
to estimate the relationship between VVA and AP exam outcomes, including an interaction term for the student’s 
grade level and the course he or she took. The first row displays the coefficient of interest: the relationship 
between VVA and the test outcome. Subsequent rows display each of the control variables in the model and the 
number of enrollments in that model. These models count number of enrollments rather than number of students 
because a student may have been enrolled in more than one course in a subject area. (Exhibits C2 to C17 present 
the results of a sensitivity test aggregating results to the student level.) 


Advanced Placement test scores 


The linear regression results estimating the relationship between Virtual Virginia and AP z-scores are in exhibits 
C54—-C58, including an interaction term for the student’s grade level and the course the student took. 
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Exhibit C54. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with a grade-by-course interaction term — Mathematics 


Estimate SE 

Virtual Virginia course -0.34 0.07 
Baseline score 0.39 0.01 
Grade 10 0.43 0.47 
Grade 11 0.41 0.43 
Grade 12 0.52 0.44 
Outcome test is calculus AB -0.06 0.05 
Outcome test is calculus BC -0.31 0.07 
Outcome test is computer science -0.32 0.07 
Outcome year is 2016/17 -0.03 0.02 
Student age -0.14 0.02 
Male 0.13 0.01 
Black/African American -0.31 0.03 
Hispanic -0.15 0.03 
Asian 0.02 0.03 
Economically disadvantaged -0.21 0.03 
Gifted 0.19 0.02 
English learner 0.10 0.05 
Receives special education services 0.12 0.04 
Pretest is algebra | 0.42 0.04 
Pretest is geometry 0.82 0.06 
Pretest year is 2013 0.09 0.04 
Pretest year is 2014 0.31 0.07 
Pretest year is 2015 0.38 0.18 
District percent rural -0.11 0.06 
Imputed for baseline score -0.01 0.05 
Grade 10*calculus AB interaction -0.95 0.60 
Grade 10*computer science interaction -0.57 0.54 
Grade 11*calculus AB interaction -1.19 0.55 
Grade 11*calculus BC interaction -0.08 0.24 
Grade 11*computer science interaction -0.61 0.48 
Grade 12*calculus AB interaction -1.20 0.56 
Grade 12*calculus BC interaction -0.03 0.26 
Grade 12*computer science interaction -0.68 0.49 
Constant -0.45 0.06 
Number of enrollments 

Virtual Virginia 296 

Face to face 29,846 

Total 30,142 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C55. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with a grade-by-course interaction term — English language arts 


Estimate SE 

Virtual Virginia course -0.06 0.07 
Baseline score 0.40 0.01 
Grade 10 -0.89 0.34 
Grade 11 -0.02 0.48 
Grade 12 0.34 0.46 
Outcome subject is English language 0.35 0.11 
Outcome year is 2016/17 -0.04 0.03 
Student age -0.11 0.01 
Male -0.04 0.01 
Black/African American -0.34 0.03 
Hispanic -0.11 0.02 
Asian 0.14 0.03 
Economically disadvantaged -0.29 0.02 
Gifted 0.39 0.02 
English learner -0.09 0.05 
Receives special education services -0.03 0.03 
Pretest year is 2013 0.06 0.03 
Pretest year is 2014 0.13 0.26 
Pretest year is 2015 1.09 0.52 
District percent rural -0.19 0.05 
Imputed for baseline score 0.26 0.02 
Grade 10*ELA interaction 2.56 0.41 
Grade 11*ELA interaction 1.84 0.41 
Grade 12*ELA interaction 1.96 0.42 
Constant -0.18 0.07 
Number of enrollments 

Virtual Virginia 291 

Face to face 41,835 

Total 42,126 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C56. Full model results for the relationship between Virtual Virginia and Advanced Placement score 
outcomes with a grade-by-course interaction term — Science 


Estimate SE 

Virtual Virginia course -0.27 0.06 
Baseline score 0.47 0.01 
Grade 10 0.29 0.56 
Grade 11 -0.07 0.55 
Grade 12 -0.04 0.55 
Outcome test is biology 0.15 0.05 
Outcome test is environmental science 0.24 0.05 
Outcome year is 2016/17 0.05 0.03 
Student age -0.12 0.01 
Male 0.18 0.02 
Black/African American -0.40 0.03 
Hispanic -0.19 0.02 
Asian -0.02 0.03 
Economically disadvantaged -0.21 0.02 
Gifted 0.28 0.02 
English learner 0.15 0.05 
Receives special education services 0.02 0.03 
Pretest year is 2013 0.00 0.03 
Pretest year is 2014 0.04 0.06 
Pretest year is 2015 0.07 0.10 
District percent rural -0.20 0.06 
Imputed for baseline score 0.06 0.03 
Grade 10*biology interaction 0.30 0.64 
Grade 10*environmental science interaction 0.04 0.65 
Grade 11*biology interaction 0.68 0.61 
Grade 11* environmental science interaction 0.56 0.63 
Grade 12*biology interaction 0.80 0.61 
Grade 11* environmental science interaction 0.71 0.63 
Constant -0.12 0.04 
Number of enrollments 

Virtual Virginia 437 

Face to face 27,859 

Total 28,296 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C57. Full model output for the relationship between Virtual Virginia and Advanced Placement score 


outcomes with a grade-by-course interaction term — History and social science 


Estimate SE 
Virtual Virginia course -0.34 0.05 
Baseline score 0.40 0.01 
Grade 10 0.27 0.17 
Grade 11 0.46 0.18 
Grade 12 0.60 0.20 
Outcome test is art history -0.14 0.09 
Outcome test is European history 0.13 0.08 
Outcome test is comparative government 0.00 0.08 
Outcome test is U.S. government -0.24 0.07 
Outcome test is human geography 0.06 0.07 
Outcome test is macroeconomics -0.41 0.09 
Outcome test is microeconomics -0.27 0.08 
Outcome test is psychology -0.01 0.07 
Outcome test is U.S. history -0.14 0.08 
Outcome year is 2016/17 0.01 0.02 
Student age -0.10 0.01 
Male 0.13 0.01 
Black/African American -0.31 0.03 
Hispanic -0.13 0.02 
Asian 0.13 0.03 
Economically disadvantaged -0.26 0.02 
Gifted 0.34 0.01 
English learner -0.03 0.03 
Receives special education services -0.01 0.02 
Pretest year is 2013 -0.02 0.02 
Pretest year is 2014 -0.01 0.04 
Pretest year is 2015 -0.01 0.08 
District percent rural -0.14 0.05 
Imputed for baseline score 0.19 0.01 
Grade 10*art history interaction -0.67 0.36 
Grade 10*European history interaction 0.03 0.28 
Grade 10*U.S. government interaction -0.46 0.38 
Grade 10*human geography interaction -0.19 0.24 
Grade 10*macroeconomics interaction -0.55 0.44 
Grade 10* microeconomics interaction -0.67 0.32 
Grade 10*psychology interaction -0.09 0.28 
Grade 10*U:S. history interaction -0.45 0.41 
Grade 11*art history interaction -0.78 0.27 
Grade 11*European history interaction 0.35 0.28 
Grade 11*comparative government and politics interaction 0.17 0.16 
Grade 11*U.S. government interaction -0.97 0.38 
Grade 11*human geography interaction -0.22 0.24 
Grade 11*macroeconomics interaction -0.69 0.42 
Grade 11*microeconomics interaction -0.99 0.29 
Grade 11*psychology interaction -0.28 0.28 
Grade 11*U:S. history interaction -0.33 0.38 
Constant -0.87 0.28 


Number of enrollments 


Virtual Virginia 2,143 
Face to face 106,974 
Total 109,117 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 
student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 

Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C58. Full model output for the relationship between Virtual Virginia and Advanced Placement score 


outcomes with a grade-by-course interaction term — Languages 


Estimate SE 

Virtual Virginia course -0.19 0.18 
Baseline score 0.28 0.02 
Grade 11 -0.28 0.22 
Grade 12 -0.19 0.22 
Outcome test is Chinese -0.19 0.17 
Outcome test is French 0.09 0.09 
Outcome test is Latin -0.02 0.14 
Outcome year is 2016/17 -0.21 0.18 
Student age -0.17 0.04 
Male -0.02 0.04 
Black/African American -0.25 0.09 
Hispanic 0.62 0.06 
Asian 0.12 0.11 
Economically disadvantaged -0.03 0.07 
Gifted 0.14 0.04 
English learner 0.36 0.07 
Receives special education services -0.28 0.08 
Pretest year is 2013 0.07 0.18 
Pretest year is 2014 0.02 0.19 
District percent rural -0.19 0.10 
Imputed for baseline score 0.33 0.06 
Grade 11*Chinese interaction 0.08 0.23 
Grade 11*French interaction 0.16 0.21 
Grade 11*Latin interaction 0.13 0.35 
Grade 12*Chinese interaction 0.09 0.25 
Grade 12*French interaction -0.05 0.24 
Grade 12*Latin interaction 0.04 0.23 
Constant -0.06 0.11 
Number of enrollments 

Virtual Virginia 136 

Face to face 3,417 

Total 3,553 


Note: In this analysis, z-scores were used as an outcome and baseline test measure, derived from AP exam scores. Z-scores were created by taking each 
student's AP exam score and subtracting the mean and dividing it by the standard deviation. Z-scores have a mean of O and a standard deviation of 1. A 
negative number indicates that the student was below the average for his or her grade level and school year, whereas a positive number indicates that a 


student was above average for his or her grade level and school year. Average z-scores for the overall population are always zero. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Advanced Placement proficiency 


The logistic regression results estimating the relationship between VVA and AP proficiency (a score of 3 or higher) 
are in exhibits C59-C62, including an interaction term for the student’s grade level and the course the student 


took. 
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Exhibit C59. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with a grade-by-course interaction term — Mathematics 


Estimate SE 

Virtual Virginia course -0.80 0.18 
Baseline score 0.87 0.03 
Grade 10 -0.33 1.70 
Grade 11 -0.38 1.70 
Grade 12 -0.05 1.71 
Outcome test is calculus AB 0.59 0.14 
Outcome test is calculus BC 0.63 0.20 
Outcome test is computer science 0.11 0.17 
Outcome year is 2016/17 -0.21 0.05 
Student age -0.29 0.05 
Male 0.28 0.03 
Black/African American -0.71 0.09 
Hispanic -0.36 0.07 
Asian 0.03 0.06 
Economically disadvantaged -0.51 0.07 
Gifted 0.42 0.04 
English learner 0.21 0.14 
Receives special education services 0.36 0.11 
Pretest is algebra | 1.09 0.14 
Pretest is geometry 1.92 0.17 
Pretest year is 2013 0.25 0.09 
Pretest year is 2014 0.95 0.20 
Pretest year is 2015 0.20 1.68 
District percent rural -0.21 0.12 
Imputed for baseline score 0.02 0.14 
Grade 10*calculus AB interaction -12.46 2.03 
Grade 10*calculus BC interaction -0.45 1.08 
Grade 10*computer science interaction -1.68 1.70 
Grade 11* calculus AB interaction -13.49 2.01 
Grade 11*calculus BC interaction -0.07 0.21 
Grade 11*computer science interaction -1.68 1.67 
Grade 12*calculus AB interaction -13.50 2.06 
Grade 12*computer science interaction -2.01 1.71 
Constant -0.79 0.17 
Number of enrollments 

Virtual Virginia 296 

Face to face 29,846 

Total 30,142 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C60. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with a grade-by-course interaction term — Science 


Estimate SE 

Virtual Virginia course -0.70 0.16 
Baseline score 1.13 0.03 
Grade 10 0.54 1.61 
Grade 11 -0.07 1.59 
Grade 12 0.03 1.60 
Outcome test is biology 2.10 0.12 
Outcome test is environmental science 1.24 0.13 
Outcome year is 2016/17 0.38 0.07 
Student age -0.28 0.04 
Male 0.45 0.04 
Black/African American -1.12 0.08 
Hispanic -0.45 0.08 
Asian -0.08 0.05 
Economically disadvantaged -0.59 0.07 
Gifted 0.64 0.05 
English learner 0.22 0.16 
Receives special education services 0.01 0.08 
Pretest year is 2013 -0.07 0.07 
Pretest year is 2014 0.02 0.17 
Pretest year is 2015 0.05 0.27 
District percent rural -0.39 0.12 
Imputed for baseline score 0.17 0.08 
Grade 10*biology interaction 0.10 1.58 
Grade 10*environmental science interaction 0.23 1.86 
Grade 11*biology interaction 0.94 1.59 
Grade 11* environmental science interaction 1.27 1.85 
Grade 12*biology interaction 1.17 1.59 
Grade 12* environmental science interaction 1.54 1.84 
Constant -0.95 0.10 
Number of enrollments 

Virtual Virginia 456 

Face to face 28,022 

Total 28,478 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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Exhibit C61. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with a grade-by-course interaction term — History and social science 


Estimate SE 
Virtual Virginia course -0.76 0.10 
Baseline score 0.94 0.02 
Grade 10 0.68 0.22 
Grade 11 1.08 0.25 
Grade 12 1.07 0.26 
Outcome test is art history -0.04 0.26 
Outcome test is European history -0.90 0.14 
Outcome test is comparative government -0.66 0.17 
Outcome test is U.S. government -0.37 0.13 
Outcome test is human geography -0.09 0.12 
Outcome test is macroeconomics -0.48 0.22 
Outcome test is microeconomics -0.18 0.20 
Outcome test is psychology 0.25 0.13 
Outcome test is U.S. history -0.61 0.14 
Outcome year is 2016/17 0.11 0.04 
Student age -0.22 0.02 
Male 0.33 0.03 
Black/African American -0.69 0.06 
Hispanic -0.31 0.05 
Asian 0.26 0.06 
Economically disadvantaged -0.56 0.05 
Gifted 0.70 0.03 
English learner -0.04 0.08 
Receives special education services -0.03 0.05 
Pretest year is 2013 -0.20 0.05 
Pretest year is 2014 -0.11 0.10 
Pretest year is 2015 -0.25 0.20 
District percent rural -0.24 0.11 
Imputed for baseline score 0.38 0.04 
Grade 10*art history interaction -16.00 1.37 
Grade 10*European history interaction -0.06 0.61 
Grade 10*comparative government and politics interaction 0.15 0.31 
Grade 10*U.S. government interaction -0.61 0.79 
Grade 10*human geography interaction -0.53 0.50 
Grade 10*macroeconomics interaction -0.66 0.72 
Grade 10*microeconomics interaction -13.80 1.13 
Grade 10*psychology interaction 0.04 0.60 
Grade 10*U.S. history interaction -0.90 1.27 
Grade 11*art history interaction -15.99 1.20 
Grade 11*European history interaction 0.49 0.60 
Grade 11*comparative government and politics interaction 0.37 0.52 
Grade 11*U.S. government interaction -1.98 0.86 
Grade 11*human geography interaction -0.49 0.49 
Grade 11*macroeconomics interaction -1.24 0.65 
Grade 11*microeconomics interaction -14.86 1.07 
Grade 11*psychology interaction -0.32 0.55 
Grade 11*U:S. history interaction -0.64 1.25 


C-65 


-16.26 1.17 


Constant 

Number of enrollments 

Virtual Virginia 2,143 

Face to face 106,974 
109,117 


Total 
Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 


Exhibit C62. Full model output for the relationship between Virtual Virginia and Advanced Placement 
proficiency outcomes with a grade-by-course interaction term — Languages 


Estimate SE 

Virtual Virginia course -0.49 0.41 
Baseline score 0.66 0.08 
Grade 11 14.68 1.53 
Grade 12 -0.31 1.24 
Outcome test is Chinese 0.04 0.85 
Outcome test is French 1.64 0.61 
Outcome test is Latin -1.40 0.25 
Outcome year is 2016/17 -2.20 0.28 
Student age -0.85 0.65 
Male -0.37 0.14 
Black/African American -0.22 0.11 
Hispanic -0.87 0.23 
Asian 1.17 0.24 
Economically disadvantaged 0.44 0.22 
Gifted -0.17 0.18 
English learner 0.21 0.15 
Receives special education services 1.34 0.38 
Pretest year is 2013 -0.46 0.25 
Pretest year is 2014 0.19 0.78 
District percent rural -13.38 1.42 
Imputed for baseline score -0.37 0.25 
Grade 11*Chinese interaction 0.60 0.23 
Grade 11*French interaction -12.75 0.76 
Grade 11*Latin interaction 1.98 0.67 
Grade 12*Chinese interaction 0.25 0.92 
Grade 12*French interaction -11.88 1.01 
Grade 12*Latin interaction 1.45 0.75 
Constant 0.56 0.79 
Number of enrollments 

Virtual Virginia 136 

Face to face 3,417 

Total 3,553 


Note: In this analysis, AP proficiency was used as an outcome. AP exam scores range from 1 to 5, where 3 is considered proficient. The outcome used in 
these analyses is dichotomous, with scores of 3 or higher scored as 1 and scores of 2 or lower scored as 0. 
Source: Authors’ analysis of administrative data provided by the Virginia Longitudinal Data System. 
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